I wanted to show how to perform a “women on the board of directors” analysis using Bayesian inference. What this means is that we need to form a “prior” on what we think the distribution of the answer could be, and then we update our prior with the data available. In this case we simplify the question we are trying to answer: given that we see a board with 3 women and 7 men (so 10 total), what is the fraction of women available for the board of directors in the general population? The reason we may want to answer this question is that then we can compare the answer to other available answers, derived other ways (say by looking at the makeup of upper level management) and see if there’s a bias.
In order to illustrate Bayesian techniques, I’ve simplified it further to be a discrete question. So I’ve pretended that there are only 11 answers you could possible have, namely that the fraction of available women (in the population of people qualified to be put on the board of directors) is 0%, 10%, 20%, …, 90%, or 100%.
Moreover, I’ve put the least judgmental prior on the situation, namely that there is an equal chance for any of these 11 possibilities. Thus the prior distribution is uniform:
The next step is to update our prior with the available data. In this case we have the data point that there a board with 3 women and 7 men. In this case we are sure that there are some women and some men available, so the updated probability of there being 0% women or 100% women should both be zero (and we will see that this is true). Moreover, we would expect to see that the most likely fraction will be 30%, and we will see that too. What Bayesian inference gives to us, though, is the relative probabilities of the other possibilities, based on the likelihood that one of them is true given the data. So for example if we are assuming for the moment that 70% of the qualified people are women, what is the likelihood that the board ends up being 3 women and 7 men? We can compute that as (0.70)^3*(0.30)^7. We multiply that by 1/11, the probability that 70% is the right answer (according to our prior) to get the “unscaled posterior distribution”, or the likelihoods of each possibility. Here’s a graph of these numbers when I do it for all 11 possibilities:
In order to make this a probability distribution we need to make sure the total adds up to 1, so we scale to get the actual posterior distribution:
What we observe is, for example, that it’s about twice as likely for 50% of women to be qualified as it is for 10% of women to be qualified, even though those answers are equally distant from the best guess of 30%. This kind of “confidence of error” is what Bayesian inference is good for. Also, keep in mind that if we had had a more informed prior the above graph would look different; for example we could use the above graph as a prior for the next time we come across a board of directors. In fact that’s exactly how this kind of inference is used: iteratively, as we travel forward through time collecting data. We typically want to start out with a prior that is pretty mild (like the uniform distribution above) so that we aren’t skewing the end results too much, and let the data speak for itself. In fact priors are typically of the form, “things should vary smoothly”; more on what that could possibly mean in a later post.
Here’s the python code I wrote to make these graphs:
Here’s the R code that Daniel Krasner wrote for these graphs:
I was looking through an old photo album (the kind where there are sticky pages and actual physical photos- it looks like an ancient technology now) and I came across one of my favorites of all time- a picture of me being embraced and supported by Cora Sadosky on one side and Barry Mazur on the other. This picture was taken in 1993 in Vancouver, where I received the Alice T. Schafer prize. It was a critical moment for me, and both of those people have influenced me profoundly. Barry became my thesis advisor; part of the reason I went into number theory was to become his student (the other part was this book).
Cora became my mathematical role model and spiritual mother. I already wrote earlier about how going to math camp when I was 14 changed my life and made me realize there is a whole community of math nerds out there and that I belonged to that nerd community. Well, Cora, whom I met when I was 21, was the person that made me realize there is a community of women mathematicians, and that I was also welcome to that world.
Actually it was something I didn’t even really want to know at the time. After all, I was happy to be a successful math undergraduate at UC Berkeley, frolicking in the graduate student lounge and partaking in tea every day at 3:00. Who cares that I was a woman? It seemed antiquated to me, almost crude, to mention my gender. When I got word that I’d won the prize, my reaction was essentially, “is there money?” (there was a bit).
And when I meet young women in math nowadays with that attitude, I am happy for them, really very happy for them. To live in that state of not caring what your gender is in mathematics is a kind of bliss, that lasts until the very moment it stops. My greatest wish for future generations of women in math is for that bliss to never stop.
And yet. I went to Vancouver and met Cora and learned about Alice Shafer and her struggles and successes as a trailblazer for women in math, and I felt really honored to be collecting an award in her name. And I felt honored to have met Cora, whose obvious passion for mathematics was absolutely awe-inspiring. She was the person who first explained to me that, as women mathematicians, we will keep growing, keep writing, and keep getting better at math as we grow older (unlike men who typically do their best work when they’re 29), and we absolutely have to maintain a purpose and a drive and fortitude for that highest call, the struggle of creation.
I kept up with Cora over the years. Every now and then she’d write to me and send me pushy little maternal notes reminding me to work hard and stay strong and productive. And I’d write to her with news of my life and my growing family and sometimes when I visited D.C. I’d meet her and we’d have lunch or dinner and talk about ideas and great books we’d read and how much we loved each other.
When I googled her this morning, I found out she’d died about 6 months ago. You can read about her difficult and inspiring mathematical career in this biography. It made me cry and made me think about how much the world needs role models like Cora.
First of all, I changed the theme of the blog, because I am getting really excellent comments from people but I thought it was too difficult to read the comments and to leave comments with the old theme. This way you can just click on the word “Go to comments” or “Leave a comment” which is a bit more self-evident to design-ignorant people like me. Hope you like it.
Next, I had a bad day today, but I’m very happy to report that something has raised my spirits. Namely, Jake Porway from Data Without Borders and I have been corresponding, and I’ve offered to talk to prospective NGO’s about data, what they should be collecting depending on what kind of studies they want to be able to perform, and how to store and revise data. It looks like it’s really going to happen!
In fact his exact words were: I will definitely reach out to you when we’re talking to NPOs / NGOs.
Oh, and by the way, he also says I can blog about our conversations together as well as my future conversations with those NGO’s (as long as they’re cool with it), which will be super interesting.
Oh, yeah. Can I get a WOOHOO?!?
In a previous post, I wrote about what I see as the cowardice and small-mindedness of the U.S. government and in particular the regulators for not demanding daily portfolios of all large investors. Of course this goes for the governments in Europe as well, and especially right now. The Economist had a good article this past Friday which attempted to quantify the results of a Greek default, but there were major holes, especially in the realm of “who owns the CDS contracts on Greek bonds, and how many are there?”. This fear of the unknown is a root cause of the current political wrangling which will probably end in a postponement of resolving the Greek situation; the question is whether the borrowed time will be used properly or squandered.
It’s ridiculous that nobody knows where the risk lies, but as a friend of mine pointed out to me last week at lunch, it probably won’t be enough to demand the portfolios daily, even if you had the perfect quantitative risk model available to you to plug them into. Why? Because if “transparency” is what the regulators demand, then “transparency” is what they would get – in the form of obfuscated lawyered-up holding lists.
In other words, let’s say a bank has a huge pile of mortgage-backed securities of dubious value on their books, but doesn’t want to accept losses on them. If they knew they’d have to start giving their portfolio to the SEC daily instead of quarterly, it would change the rules of the game. They’d have to hide these holdings by pure obfuscation rather than short-term month- or quarter-end legal finagling. So for example, they could invest in company A, which invests in company B, which happens to have a bunch of mortgage-backed securities of dubious value, but which is too small to fall under the “daily reporting” rules. This is just an example but is probably an accurate portrayal of the kind of thing that would happen with enough lead time and enough lawyers.
What we actually want is to set up a system whereby banks and hedge funds are motivated to be transparent. Read this as: will lose money if they aren’t transparent, because that’s the only motivation that they respond to.
In some sense, as my friend reminded me, we don’t need to worry about hedge funds as much as about banks. This is because hedge funds do their trades through brokerages, which force margin calls on trades that they deem risky. In other words, they pay for their risk through margins on a trade-by-trade, daily basis. If you are thinking, “wait, what about LCTM? Isn’t that a hedge fund that got away with murder and almost blew up the system and didn’t seem to have large margins in place?” then the answer is, “yeah but brokers don’t get fooled (as much) by hedge funds anymore”. In other words, brokers, who are major players in the financial game, are the policemen of hedge funds.
There are two major limits to the above argument. Firstly, hedge funds purposefully use multiple brokers simultaneously so that nobody knows their entire book, so to the extent that risk of portfolio isn’t additive (it isn’t), this policing method isn’t complete. Secondly, it is only a local kind of risk issue- it doesn’t clarify risk given a catastrophic event (like a Greek default), but rather a more work-a-day “normal circumstances” market risk.
Even so, what about the banks? Are there any brokers measuring the risk of their activities and investments? Since the banks are the brokers, we have to look elsewhere… I guess that would have to be at the government, and the regulators themselves, maybe the FDIC… in any case, people decidedly not players in the financial game, not motivated by pay-off, and therefore not prone to delving into the asperger-inspiring details of complicated structured products to search out lies or liberal estimates.
The goal then is to create a new kind of market which allows insiders to bet on the validity of banks’ portfolios. You may be saying, “hey isn’t that just the stock price of the bank itself?”, and to answer that I’d refer you to this article which does a good job explaining how little information and power is actually being exercised by stockholders.
I will follow up this post with another more technical one where I will attempt to describe the new market and how it could (possibly, hopefully) function to motivate transparency of banks. But in the meantime, feel free to make suggestions!
This is a co-post with FogOfWar.
Here’s an interesting article about how many board of directors for S&P500 companies consist entirely of men. Turns out it’s 47. Well, but we’d expect there to be some number of boards (out of 500) which consist entirely of men even if half of the overall set of board members are women. So the natural question arises, what is the most likely actual proportion of women given this number 47 out of 500?
In fact we know that many people are on multiple boards but for the sake of this discussion let’s assume that there’s a line of board seekers standing outside waiting to get in, and that we will randomly distribute them to boards as they walk inside, and we are wondering how many of them are women given that we end up with 47 all-men boards out of 500. Also, let’s assume there are 8 slots per board, which is of course a guess but we can see how robust that guess is by changing it at the end.
By the way, I can think of two arguments as to why the simplification that nobody is on multiple boards argument might skew the results. On the one hand, we all know it’s an old boys network so there are a bunch of connections that a few select men enjoy which puts them on a bunch of boards, which probably means the average number of boards that a man is on, who is on at least one board, is pretty large. On the other hand, it’s also well-known that, in order to seem like you’re diverse and modern, companies are trying to get at least a token woman on their board, and for some reason consider the task of finding a qualified woman really difficult. Thus I imagine it’s quite likely that once a woman has been invited to be on a board, and she’s magically dubbed “qualified,” then approximately 200 other boards will immediately invite that same woman to be on their board (“Oh my god, they’ve actually found a qualified woman!”). In other words I imagine that the average number of boards a given woman is on, assuming she’s on one, is probably even higher than for men, so our simplifying assumptions will in the end be overestimating the number of women on boards. But this is just a guess.
Now that I’ve written that argument down, I realize another reason our calculation below will be overestimating women is this concept of tokenism- once a board has one woman they may think their job is done, so to speak, in the diversity department. I’m wishing I could really get my hands on the sizes and composition of each board and see how many of them have exactly one woman (and compare that to what you’d expect with random placement). This could potentially prove (in the sense of providing statistically significant evidence for) a culture of tokenism. If anyone reading this knows how to get their hands on that data, please write!
Now to the calculation. Assuming, once more, that each board member is on exactly one board and that there are 8 people (randomly distributed) per board, what is the most likely percentage of overall women given that we are seeing 47 all-male boards out of 500? This boils down to a biased coin problem (with the two sides labeld “F” and “M” for female and male) where we are looking for the bias. For each board we flip the coin 8 times and see how many “F”s we get and how many “M”s we get and that gives us our board.
First, what would the expected number of all-male boards be if the coin is unbiased? Since expectation is additive and we are modeling the boards as independent, we just need to figure out the probability that one board is all-male and multiply by 500. But for an unbiased coin that boils down to (1/2)^8 = 0.39%, so after multiplying by 500 we get 1.95, in other words we’d expect 2 all-male boards. So the numbers are definitely telling us that we should not be expecting 50% women. What is the most likely number of women then? In this case we work backwards: we know the answer is 47, so divide that by 500 to get 0.094, and now find the probability p of the biased coin landing on F so that all-maleness has probability 0.094. This is another way of saying that (1-p)^8 = 0.094, or that 1-p is 0.744, the eighth root of 0.094. So our best guess is p = 25.6%. Here’s a table with other numbers depending on the assumed size of the boards:
If anyone reading this has a good sense of the distribution of the size of boards for the S&P500, please write or comment, so I can improve our estimates.
A nerd friend of mine kindly rewrote my python scripts in R and produced similar looking graphs. I downloaded R from here and one thing that’s cool is that once it’s installed, if you open an R source code (ending with “.R”), an R console pops up automatically and you can just start working. Here’s the code:
gdata <- read.csv('large_data_glucose.csv', header=TRUE)
#We can open a spreadsheet type editor to check out and edit the data: edit(gdata) #Since we are interested in the glucose sensor data, column 31, but the name is a bit awkward to deal with, a good thing to do is to change it: colnames(gdata) <- "GSensor" #Lets plot the glucose sensor data: plot(gdata$GSensor, col="darkblue") #Here's a histogram plot: hist(gdata$GSensor, breaks=100, col="darkblue")
#and now lets plot the logarithm of the data: hist(log(gdata$GSensor), breaks=100, col="darkblue")
And here are the plots:
One thing my friend mentions is that R automatically skips missing values (whereas we had to deal with them directly in python). He also mentions that other things can be done in this situation, and to learn more we should check out this site.
R seems to be really good at this kind of thing, that is to say doing the first thing you can think about with data. I am wondering how it compares to python when you have to really start cleaning and processing the data before plotting. We shall see!
I want to describe the culture of working at D.E. Shaw during the credit crisis, so from June 2007 to June 2009, because I think it’s emblematic of something that most news articles and books written about hedge funds really miss out on when they fixate on the average I.Q. of the people working there, which is in the end a distraction and nothing more, or the bizarre or quirky personalities that exist there, which is only idiosyncratic and doesn’t explain anything deeply.
I promised myself I’d put focus on the following phrase, which struck me down when I first heard it used and still makes me shake my head, namely the concept of “dumb money.” The phrase was tossed around constantly and cleverly, and really, to understand what it means inside the context of the hedge fund culture, is to understand the culture. So I’ll try to explain it. First a bit of context.
Most of the quants at D.E. Shaw were immigrant men. In fact I was the only woman quant when I joined, and there were quite a few quants, maybe 50, and I was also one of the only Americans. What nearly all these men had in common was a kind of constant, nervous hunger, almost like a daily fear that they wouldn’t have enough to eat. At first I thought of them as having a serious chip on their shoulder, like they were the kind of guy that didn’t make the football team in high school and were still trying to get over that. And I still think there’s an element of something as simple as that, but it goes deeper. One of my colleagues from Eastern Europe said to me once, “Cathy, my grandparents were coal miners. I don’t want my kids to be coal miners. I don’t want my grandchildren to be coal miners. I don’t want anybody in my family to ever be a coal miner again.” So, what, you’re going to amass enough money so that no descendent of yours ever needs to get a job? Something like that.
But here’s the thing, that fear was real to him. It was that earnest, heartfelt anxiety that convinced me that I was really different from these guys. The difference was that, firstly, they were acting as if a famine was imminent, and they’d need to scrounge up food or starve to death, and secondly, that only their nuclear family was worth saving. This is where I really lost them. I mean, I get the idea of acts of desperation to survive, but I don’t get how you choose who to save and who to let die. However, it was this kind of us-against-them mentality that prevailed and informed the approach to making money.
Once you understand the mentality, it’s easier to understand the “dumb money” phrase. It simply means, we are smarter than those idiots, let’s use our intelligence to anticipate dumb peoples’ trades and take their money. It is our right as intelligent, imminently starving people to do this. Chasing dumb money can take various forms, but is generally aimed at anticipating lazy fund managers: if you know that they always wait until Friday afternoon to balance their books, or that they wait until the end of the month, or that they are required to buy certain kinds of things, you can anticipate their trades, make them yourself a bit before they do, thereby forcing them to pay more, and getting a nice little profit for yourself. In short this works in general, since statistically speaking the anticipated trade wasn’t driving up the intrinsic value of the underlying, but rather was being affected by trade impact for a short amount of time. If we can anticipate big trades by lots of dumb money, then the short-term market impact will be large enough and last long enough to buy in beforehand and sell at the top, while it still lasts, assuming there’s sufficient liquidity. The subtext of taking dumb money, going back to the football team issue, is: if we don’t somebody else will, and then we will feel like fools for not doing it ourselves.
To tell you the truth, I was completely naive when I went to work there. I had kind of accepted the job because I wanted to be a business woman, wanted a brisk pace after the agonizing slowness of academics, and I had really no moral judgment on the concept of a hedge fund; I thought it was morally neutral, at worst a scavenger on the financial system, like a market maker or someone who provides insurance for something. Well I’ve decided it’s more like a leech.
Getting to the part about actually working with Larry Summers. I did work on a couple of his ideas, although in order not to get sued I can’t be detailed about what his ideas were. And I had various meetings with him and a bunch of managing directors. One thing I remember about these meetings was the eery way the managing directors seemed intimidated by him, even though behind his back they kind of scoffed at the possibility that he could actually offer good modeling ideas. It was basically a publicity stunt, or at least rumored to be, to have him work there. It was after he had gotten pushed out of the Presidency at Harvard for talking out of his ass about women in math, and yes it was a bit surreal to be the only woman quant in the place, and to be working on his project considering that. Since I am pretty much never intimidated for some reason, I had no problem. He kept on grilling me about various things to try and I kept explaining what I’d done and how I’d already thought of that. It was fine, pretty combative and pushy, but actually kind of fun. I really have nothing to say about him treating me differently because I was a woman.
But when I think about that last project I was working on, I still get kind of sick to my stomach. It was essentially, and I need to be vague here, a way of collecting dumb money from pension funds. There’s no real way to make that moral, or even morally neutral. There’s no way to see that as scavenging on the marketplace. Nope, that’s just plain chasing after dumb money, and I needed to quit. I still don’t know if that model went into production.
FogOfWar has kindly offered the background below on the OTC market and an analogy with the bond market, inspired by this recent article describing the latest round of watering-down of derivatives regulations. The bottomline for me is that whenever you see people using the phrases “needlessly tying up capital that would otherwise be used to create jobs and grow the economy,” “would damage America,” or especially an emphasis on “U.S. firms,” it probably means they are trying to engender a local nationalistic fervor to camouflage a very basic greedy instinct. Here’s the background:
OTC derivatives, by definition, are not traded on an open exchange, but are entered into between two parties in a private transaction. We can use JPMorgan and United Airlines as a running example. United has some risk it has that it wants to hedge. Or maybe some banker has convinced United that they should be hedging a risk that they didn’t know they had until the banker showed up to tell them about it.
For some simple things, United could just go to an exchange (a stock market, but not limited to stocks). So, for example, United could buy a future on oil prices to lock in its cost of oil over the next year. The problem is that there actually aren’t that many different contracts traded on exchanges, and the risks don’t usually fit neatly into the contracts that are there to buy. There’s a whole chapter on how to get a best-possible hedge in this situation in Derivatives 101 (and probably a whole class after that and people who make a living doing it in real life). So you could ‘dirty hedge’ (do an imperfect hedge), or you could go to JPMorgan and ask for an exact hedge.
JPMorgan is happy to give you the hedge and either delta hedge out the risk and/or match against offsetting risk they have on their books (or even use the opportunity to take a speculative position they were thinking about anyway). The key point is that JPMorgan quotes you a price, but it isn’t a price on a transparent market–it’s just whatever price they think you’ll pay. If United is smart, they’ll farm out the hedging for a bunch of bids from different banks and try to get the best price, but they’ll never actually know if they got ripped off or not, because they don’t see how much it actually costs JPMorgan to cover that risk internally.
There are some areas where the hedges are common enough and enough people offer them OTC that the profit margins are pretty low (simple interest rate swaps are a good example). However, there is also a lot of money to be made from ripping off dumb customers like United when they wander outside of these areas into other areas where they get crap pricing. This is how derivatives trading desks make their bonus.
And that’s why JPMorgan cares about this legislation. They want to keep ripping off the United Airlines of the world, and if the government makes United go to an actual exchange with open prices, there’ll be competition and the profit margin will shrink. Adding a margin requirement is a bit more wonky, but at the end JPMorgan doesn’t like it because it might drive United to an exchange and away from an OTC derivatives trade with JPMorgan.
It may go without saying, but Jamie Dimon, JPMorgan, GS, BofA, etc. do not give a shit whatsoever about United, Shell, Alcoa, or any other corporate. They just want the profits from their OTC derivatives trading desk to keep rolling in–profits that come off the backs of their customers–and they’ll say whatever garbage they think Congress and the Agencies will swallow to keep the trades rolling.
Felix Salmon wrote all this up a ways back when Barney Frank was caving to the investment banks and putting the end-user exception into Dodd-Frank to begin with. That was around the time my opinion of Barney Frank went from “rock star” to “big fat pussy”. The history (Salmon honed in on this) tells the story in the world of bond trading–what follows is a very general overview from memory:
Once upon a time, if a corporate wanted to buy bonds, they went to their investment bank. They didn’t see exchange-listed prices, and maybe they got a few quotes to try to get good pricing, but at the end of the day, much like the OTC derivatives market described above, they either had to take a price offered by a bank or not.
Then the government came in and said bonds should be traded on open exchanges (with bid and ask prices available for participants to see). The banks said it would destroy the market, they said the corporates would suffer, they said the markets would move overseas, they probably said it would “hurt America” to do this. All of exactly the same horse shit Jamie Dimon and the banks are saying now about moving derivatives to exchanges.
Well, bond trading got moved to exchanges and exactly none of the things the banks warned about actually happened. Instead, the thing all of the banks were secretly fearing did happen: customers got good execution at lower prices and bank profit margins in the bond business slowly collapsed over time to a fraction of what they were back in the OTC-bond days. Go figure.
I wanted to tell you about my experience a few months back sitting on a “non-academic career” panel at a women’s math conference at IPAM on the UCLA campus*.
The panel consisted of four women who have Ph.Ds in math and had left academics for other things. I was the only person from finance represented- the other women worked for government sponsored research agencies (NSA, Aerospace Corporation, and Los Alamos National Labs). In fact considering the fact that I was unemployed you could say I was a bad role model but I decided to attend anyway because I thought I could contribute- and after all, being unemployed is part of life, especially outside of academics. The audience consisted of about 50 women (and one man who never announced himself but I think worked for the NSF) who were either finishing up their Ph.Ds or had recently gotten them and were in post-docs – the perfect moment for a little inspirational speech. The panelists were given about 10 minutes each to talk about their experiences and then the audience had a chance to ask questions. Here’s (more or less) what I said to them.
Hi, I’m your unemployed role model. I thought of not coming here today since, after all, I’m unemployed, and what kind of role model does that make me? Actually it makes me a good one, and here’s why. That job I left wasn’t good enough for me. I didn’t get fired, I quit (although plenty of great people I know have been laid off so that’s no proof of anything). The truth is, I deserve a job that I really like, where I’m challenged to grow and to learn and to do my best and I’m rewarded for doing so. After all, I have a super power, which is mathematics. So the reason I’m saying this is that you do too. All of you have a superpower, which is mathematics. You all deserve to work at good jobs that you actually enjoy- and if the jobs you have turn out to be bad, or if the become bad for some reason, then quit! Get another one! Get a better one! I actually got a job offer on the plane over here yesterday (true!). I know I’m going to get a good job, even in this economy, because I can do something other people actually regard as magical. Mathematical training and thinking is something that everybody needs and not everybody can achieve, so remember that. Never feel stuck. This is not to say that the specific training you have right now is sellable on the open market, but since you’re a mathematician the one thing you can count on being good at is learning new stuff. So if you decide to change fields, get ready to roll up your sleeves and work your butt off to learn the necessary stuff, but be sure that you can do it and that it will be really important to the people you work for. And if it isn’t, or if you don’t think your work is being appreciated, go get a better job. Thanks!
By the way, I thought I should give you a wee bit of context for the above speech. It’s my opinion that one of the main reasons you don’t see as many women as men in great positions in the sciences is that, just around this point in their careers, women lack the confidence to sell themselves strongly in the job market. In fact the confidence problem exists earlier- I don’t know how many times I’ve seen (at MIT and then at Barnard/Columbia) a woman in my class crying over a grade of 94 (out of 100) in office hours while at the same time I talk to a man who got a 64 who says, “no problem I’ll ace the final”. However, once the grades stop and the field becomes more about confidence in yourself than in other peoples’ grades, women often flounder.
Just as a measurement of how much women need to hear this kind of stuff, I must have been approached by nearly all the women in the audience on a person-by-person basis, as well as emailed by quite a few thanking me for my inspirational words. Many of them expressed relief and astonishment that their accomplishments are actually worthy of a good job. These women need to hear this stuff- not to become arrogant but just to know that they really have something to offer. Also many of them are married or engaged and many of them have small kids, so they also need to prioritize themselves, which is again something they have trouble with.
And one more thing: I love encouraging young men in math too.
* This has been posted at my friend Chelsea’s blog already.
A friend of mine has type I diabetes, and lots of data (glucose levels every five minutes) from his monitor. We’ve talked on and off about how to model future (as in one hour hence) glucose levels, using information on the current level, insulin intake, and carb intake. He was kind enough to allow me to work on this project on this blog. It’s an exciting and potentially really useful project, and it will be great to use as an example for each step of the modeling process.
To be clear: I don’t know if I will be able to successfully model glucose levels (or even better be able to make suggestions for how much insulin or carbs to take in order to keep glucose levels within reasonable levels), but it’s exciting to try and it’s totally worth a try. I’m counting on you to give me suggestions if I’m being dumb and missing something!
I decided to use python to do my modeling, and I went to this awesomely useful page and followed the instructions to install python and matplotlib on my oldish mac book. It worked perfectly (thanks, nerd who wrote that page!).
The data file, which contains 3 months of data, is a csv (comma separated values) file, with the first line describing the name of the values in the lines below it:
Index,Date,Time,Timestamp,New Device Time,BG Reading (mg/dL),Linked BG Meter ID,Temp Basal Amount (U/h),Temp Basal Type,Temp Basal Duration (hh:mm:ss),Bolus Type,Bolus Volume Selected (U),Bolus Volume Delive\ red (U),Programmed Bolus Duration (hh:mm:ss),Prime Type,Prime Volume Delivered (U),Suspend,Rewind,BWZ Estimate (U),BWZ Target High BG (mg/dL),BWZ Target Low BG (mg/dL),BWZ Carb Ratio (grams),BWZ Insulin Sens\ itivity (mg/dL),BWZ Carb Input (grams),BWZ BG Input (mg/dL),BWZ Correction Estimate (U),BWZ Food Estimate (U),BWZ Active Insulin (U),Alarm,Sensor Calibration BG (mg/dL),Sensor Glucose (mg/dL),ISIG Value,Dail\ y Insulin Total (U),Raw-Type,Raw-Values,Raw-ID,Raw-Upload ID,Raw-Seq Num,Raw-Device Type 1,12/15/10,00:00:00,12/15/10 00:00:00,,,,,,,,,,,,,,,,,,,,,,,,,,,,,28.4,ResultDailyTotal,"AMOUNT=28.4, CONCENTRATION=null",5472682886,50184670,236,Paradigm 522 2,12/15/10,00:04:00,12/15/10 00:04:00,,,,,,,,,,,,,,,,,,,,,,,,,,,120,16.54,,GlucoseSensorData,"AMOUNT=120, ISIG=16.54, VCNTR=null, BACKFILL_INDICATOR=null",5472689886,50184670,4240,Paradigm 522 3,12/15/10,00:09:00,12/15/10 00:09:00,,,,,,,,,,,,,,,,,,,,,,,,,,,116,16.21,,GlucoseSensorData,"AMOUNT=116, ISIG=16.21, VCNTR=null, BACKFILL_INDICATOR=null",5472689885,50184670,4239,Paradigm 522
I made a new directory below my home directory for this file and for the python scripts to live, and I started up python from the command line inside that directory. Then I opened emacs (could have been TextEdit or any other editor you like) to write simple script to see my data.
A really easy way of importing this kind of file into python is to use a DictReader. DictReader is looking for a file formatted exactly as this file is, and it’s easy to use. I wrote this simple script to take a look at the values in the “Sensor Glucose” field (note there are sometimes gaps and I had to decide what to do in that case):
And this is the picture that popped out:
I don’t know how easy it is to see this but there are lots of gaps (when there’s a gap I plotted a dot at -1, and the line at -1 looks pretty thick). Moreover, it’s clear this data is being kept in a pretty tight range (probably good news for my friend). Another thing you might notice is that the data looks more likely to be in the lower half of the range than in the upper half. To get at this we will draw a histogram of the data, but this time we will *not* fill in gaps with a bunch of fake “-1″s since that would throw off the histogram. Here are the lines I added in the code:
And this is the histogram that resulted:
This is a pretty skewed, pretty long right-tailed distribution. Since we know the data is always positive (it’s measuring the presence of something in the blood stream), and since the distribution is skewed, this makes me consider using the log values instead of the actual values. This is because, as a rule of thumb, it’s better to use variables that are more or less normally distributed. To picture this I replace one line in my code:
skip_gaps_datalist.append(log(float(row["Sensor Glucose (mg/dL)"])))
And this is the new histogram:
This is definitely more normal.
Next time we will talk more about cleaning this data and what other data we will use for the model.
How freaking cool is this?! I signed up today and wrote to the founder, Jake Porway. He seems fantastic. I’m very excited about his project and how we (meaning you and me, kind reader) can help use our data scientist hats to help NGOs think about what data to collect and how to analyze it once they have it. Please consider signing up!
I’m delighted to have my first guest blogger!
“FogOfWar” (named after the documentary) is someone I’ve known for some time who comes from a mathy background, with a detour through accounting, tax & law winding up in banking (not as a quant). FOW & I have jammed finance policy many times and we tend to agree on a lot of things–I hope it will bring a “what really happens on the ground” perspective to thoughts about modeling as well as some useful insight into some of the technical rules (like accounting) that can matter a lot. Here’s his post:
The NYT ran an article on tax repatriation yesterday. Often, as someone in the industry, these articles can be infuriating for their lack of accuracy, misdirection or imprecision. In this case, however, my hat is off to the NYT for some damn fine traditional journalism. They’ve taken a fairly complicated issue (one I happen to know more than a little about), understood the core points in play and laid them out in an interesting, informative and readable article. Yes, it really is as bad as they make it out to be.
The “repatriation holiday” makes my vague-and-unofficial list of “10 worst tax ideas out there”. Unfortunately, every bad idea ultimately finds its way to Congress & this one is back for seconds. The NYT article lays out the case well, but here’s are two additional reasons on why this idea seems to have lasting appeal, which come in the form of catchy phrases:
“The money is trapped overseas”
We all know what “money”, “trapped” and “overseas” mean, and we can form an immediate idea of how this would be a bad thing, and how freeing that trapped money and bringing it back to the US would be good for the economy. Thus we get the inference: “The money is trapped overseas, and if we could bring it back it would create jobs.” Unfortunately, the second half of the second sentence is completely false. A more accurate sentence would be “The money is trapped overseas, and if we could bring it back corporations would pay slightly larger dividends this year, but not create any jobs or invest in any US plants that weren’t already in their strategic planning.” Doesn’t have quite the same ring to it…
Not nearly as catchy as the first phrase, and uses two words for which most people don’t have a quick definition (at least not when paired together). Relevant, however, and a quick wonkish example with illustrate the thrust:
Let’s take a hypothetical US company, called (just to pick a name at random) “Lehman Holdings”. Lehman Holdings has assets claimed at $900 on its books and debt of $800. Lehman Holdings also owns 100% of another company, who we’ll call “Lehman UK”, which has assets claimed at $100 on its books and no debt. So, at first blush, one might think that Lehman has a 20% equity buffer: $1,000 of assets and $800 of debt (or a 4:1 debt ratio). This is nice easy math, which happens to be wrong in practice. The hidden assumption is that the people who loaned Lehman Holdings $800 can get access to all $1,000 of assets. They certainly can access the $900 of assets (or whatever they’re worth by the time bankruptcy hits), but the UK subsidiary is subject to UK bankruptcy rules, not US bankruptcy rules. Thus, when US creditors try to pull the $100 of assets out of the UK, they may find it’s more difficult than they anticipated (international bankruptcy gets sticky fast). Perhaps they could sell the stock of the subsidiary, but in real life that would involve untangling a whole host of interconnected contractual arrangements between Lehman Holdings and Lehman UK, which could take years. Not to mention the fact that to pull the $100 back, they’d have to pay ($35) in US taxes, so really there may be only $65 net to work with (other facts could zero out the tax bill). Probably in the end they can get the $100 of assets ($65 post taxes), but it can mean a significant time delay, and when you’re dealing with an imminent default, delay in action can translate to financial loss.
So, for all of these reasons, having $100 in a subsidiary isn’t worth quite the same thing as having $100 in the parent company. The fancy name for this is “structural subordination”, a term used by the credit rating agencies. So, if you’re a tech or pharma company with many billions of USD in your tax-shelter Irish/Dutch/Singapore subsidiaries, this can become a problem for your credit rating (which can impact your cost of borrowing). It’s probably not the primary reason for the lobbying efforts on tax repatriation, but it is definitely a factor, as the ($35) in tax is what’s preventing Holdings in the above example from pulling the $100 out of UK.
After the credit crisis hit we all realized that there’s a lot more risk out there than can be described by trailing volatility measures. Once I decided to leave the hedge fund world, I was thinking about working for the “other side,” namely to help quantify risk and/or work on the side of the regulators. I applied to the SEC, the New York Fed, and Riskmetrics, a software company which had a good reputation. I never heard from the Fed, and the SEC didn’t seem to have something for me, but I landed a job at Riskmetrics.
I figured it this way: if you work on a risk in a good way, if you make a better risk model, then you can at least argue you are improving the world. If you are instead making a bad risk model, and you know it, then you’re making the world a worse, riskier place. For example if you are working for a rating agency and get paid to ignore signs of riskiness, then that would be the not improving the world kind.
I really enjoyed my job, and after some months I was put in charge of “risk methodology,” which meant I got to think about how to quantify risk and why. I worked on our credit default model, which was super interesting, and I got to talk to the head trader of one of the biggest CDS trading desks regularly to understand the details of the market. In fact many of the biggest hedge funds and banks and pension funds send their portfolios daily to companies such as Riskmetrics to get overnight assessments of the riskiness of their portfolios. Bottomline is that my job kind of rocked, but it didn’t last forever; we were acquired soon after that by a company which didn’t offer me the same kind of position and I left pretty soon.
Here’s an article that very clearly articulates some of the problems in the field of quantitative risk. In my opinion it doesn’t go far enough with respect to their last point, or maybe it misses something, where they talk about “forecasting extreme risks.” This refers to the kind of thing that happens in a crisis, when all sorts of people are pulling out of the market at the same time and there are cascading, catastrophic losses.
What gets to me about this is that everyone talks about moments like these as if they can’t be modeled, but of course they can be, to a limited extent. Namely, although we don’t know what the next huge crisis will be, there are a few obvious candidates (like the Greek, Portuguese, Irish, or U.S. defaulting on their debt) which we should be keeping an eye on to the best of our quantitative abilities. Many of the “panic” situations (like the mortgage-backed securities debacle) were pretty obvious risks weeks or months in advance of their occurring, but people just didn’t know how to anticipate the consequences. That’s fine for a given individual trader but shouldn’t be true for the government.
I think the first step should be to compile a longish list of possible disaster scenarios (include the ones we’ve already seen happen) and decide what the probability of each scenario is- these probabilities can be updated each week by a crew of economists or what have you. Secondly and separately, set up a quantitative model which tries to capture the resulting cascade of consequences that each scenario would create; this would be complicated and involve things like guessing the losses at which hedge funds start liquidating their books, but should be aided by amassing huge amounts of information of the underlying portfolios of the largest institutions.
In my opinion the regulators have made a huge mistake in the past three years by _not_ insisting on getting the entire portfolio from every major hedge fund and bank every night (which from above we know is possible for them since they already send them to Riskmetrics-like software companies, although I’ve read articles where they claim this would be way too onerous a task) and, with that deep information, model the effect of a crisis scenario from our above list; how would it affect the bond market? The CDS market? The model which already exists at quantitative hedge funds now, which measures the impact and decay on trades, is a great start. Moreover, this model is not impossible to train (i.e. the actual coefficients inside the model’s formulas aren’t that hard to estimate), in fact it wouldn’t be that big a deal if we had as much data as I’m talking about. To me it’s unbelievable that we aren’t getting this portfolio information every day (or even intraday) and creating a “systemic impact model,” because it would clearly make us better prepared for future events (although not of course perfectly prepared) and no hedge fund or bank could argue that we shouldn’t be worried – it should be one of the costs of doing business on Wall Street.
If you’re anything like me, you eat up the news on the Greek situation whenever and wherever you can. It’s like watching a slow-motion train wreck that takes years to hit. No, even better, it’s like this:
Imagine there’s a family that you know as self-absorbed, undisciplined, and indulgent, especially with their kids- they let their kids watch too much TV, they give their kids every gadget they can’t really afford, flat-screen TVs on credit, they stay up too late, eat crap food, they bribe their kids to like them, bringing them presents after every trip. It borders on neglect, for God’s sake, and it will come back to haunt them, you think to yourself. Then imagine seeing them in a crowded restaurant with their kids, older now, and utterly obnoxious and lazy and entitled, screaming at the top of their lungs that whatever the complaint is, it’s definitely not their fault, it’s their stinking parents’ fault, and why should they get a job. It’s an obnoxiously satisfying scene to watch as an exhausted parent who has been sure to feed their kids broccoli and have their kids tucked in by 9 with their homework done and their backpacks ready for school the next day.
But here’s the thing, I kind of have to side with the spoiled kids. I mean, it is the parents’ fault if they’ve completely spoiled their kids. As bratty as the kids are, you really can’t blame them on this until they are rational adults.
In summation, Greece is the European version of the Kardashians.
Here’s an article which kinds of proves my point. The politicians have spoiled the Greeks for so long, by buying votes with do-nothing government jobs, and simply ignoring the state of the deficit and anything involving money or taxes (mostly because the politicians themselves are the worst of the tax-evaders and don’t want to rock the boat), that the people living there are looking anywhere but at themselves for where the problem lies. In other words, a completely backwards-looking approach with no forwards-looking solution in mind. They are that kid at that restaurant, somewhere in late adolescence but not quite adults.
Another aspect of this crisis is the enormous disconnect between the economists and bankers on the one hand, who have absolute certainty that the banking system must be kept functional at any cost, and the actual people living in a country on the other hand, who don’t want to pay for the mistakes of the rich bankers. What makes this gulf so wide? It’s wide in any country actually, but in Greece you have the extra layer of spoiled entitlement. I’ll talk about this disconnect in my second post about working at D.E. Shaw, where I experienced it first-hand.
After quite a bit of feedback (love feedback!) I’ve decided to add to this post because I think I was too glib and didn’t make my point well. First, let me be clear that I don’t think that the Greek workers are spoiled. I have a lot of compassion for the working people of Greece- especially the youth. The young people of Greece have a broken system, filled with closed guilds, high unemployment, and corrupt politicians. I am extremely empathetic to their plight and if I were them I’d be protesting in the streets too. What I mean to get across with the spoiled kid thing is that spoiling kids really is neglect and really is the fault of the authorities, and it sets up someone to fail and it gives them no tools to correct systemic mistakes. In this analogy I’m trying to point out that the political class has neglected its people and its duty to create a working system. They have done nothing for those young people, and now they are trying to make inside deals with the European bankers and don’t seem to understand why the actual working (or unemployed) people of Greece don’t see why this is a great opportunity.
So a friend of mine came over last night and he recently became a data scientist in a New York startup too. In fact we have an eery number of things in common, although he only considered working in finance but didn’t actually go through it. It was pretty awesome to see him.
He was also pretty into the idea of this blog and making quantitative techniques more open-source and collaborative. And with that goal in mind he sent me these links:
So what do you guys say? Should we work on something together on this blog that may actually help the world/ make us some prize money? That would be filthy good.
I’m also hoping to get this guy to make a guest post on some quantitative techniques he wants to add to my list. Please comment if you have more suggestions! I will start writing about the list topics very soon.
One exciting goal I have for this blog is to articulate the basic methods of quantitative modeling, followed by, hopefully, collaborative real-time examples of how this craft works out in given examples. Today I just want to outline the techniques, and in later posts I will follow up with a post which goes into more detail on one or more points.
- Data cleaning: bad data (corrupt) vs. outliers (actual data which have unusual values)
- In sample/ out of sample data
- Predictive variables: choosing and preparing which ones and how many
- Exponential down-weighting of “old” data
- Remaining causal: predictive vs. descriptive modeling
- Regressions: linear and multivariate with exponentially down-weighted data
- Bayesian priors and how to implement them
- Open source tools
- When do you have enough data?
- When do you have statistically significant results?
- Visualizing everything
- General philosophy of avoiding fitting your model to the data
For those of you reading this who know a thing or two about being a quant, please do tell me if I’ve missed something.
I can’t wait!
After I had been working at D.E. Shaw for a few months, I was asked by the American Mathematic Society to write an expository article on leaving academics for finance. Here’s what I wrote. It was infinitely vetted by the legal department, and they removed a bunch of stuff- by the time they approved it I couldn’t remember why I had wanted to write it in the first place. Oh yeah, something about answering a bunch of questions that math grad students kept asking me. The one edit I refused to budge on, I remember, was that they objected to the word “rich” in the sentence “However, it is clear that if you stay in finance for long enough, and are successful, you do become rich”. They wanted change the word to “wealthy”. As if that was going to soften the blow to the poor suckers who weren’t privileged enough to work at this holy place.
Ever since it was published, I’ve wanted to write a second edition. It would go something like this (taken from a letter I wrote to a friend recently who is applying to another hedge fund):
I actually never really intended to stay in finance, it was just the only “real job” I could get with my number theory skills. In the end I decided I wanted to work at a startup and there are more internet startups than finance ones. The truth is, there are a bunch of jerks
in finance, very likely due to the amount of money floating around, and I noticed a correlation with the size/age of the company and the douchebagginess of the “leaders” of the firms. I don’t know alot about ****** but word on the street is that they are huge douchebags. On the other hand, I myself don’t regret working with douchebags for four years, because it thickened my skin quite a bit (and in particular made me realize how impotent and feeble the academic douchebags are in comparison) and made me strive for something better. Although to be honest it sometimes really sucked.
I could sum it up pretty well thus: people who are successful for a while think they know everything. People who are rich think they are always right. People who are both successful and rich are absolutely incredible douchebags. It seems like a law of nature (i.e. I can only assume that if I ever become rich and successful I will also become a douchebag. One more reason not to be wishing too hard for things like that.).
So instead I work for *pretty good* money (better than I’d have gotten in academics but not as good as at DE Shaw) and I enjoy things like oatmeal in the morning, biking to work on the bike path, my incredible adorable macho developer colleagues, a really cool hands-off boss, and a bunch of awesome karaoke-loving beer-drinking coworkers who think I have special powers since I can do math. Oh, and the possibility that someday my numerous stock options in this startup may make me a douchebag someday.
I just want to add that, of course, not everyone I worked with at D.E. Shaw is a douchebag, not even all the leaders. In fact I still have many friends from there. But it’s definitely not a random cut of the population, and I would have to believe that people in it would agree with that (and would say it’s worth it).
In part 2 of this post I will talk about what specifically made me decide to leave the hedge fund industry.
Let me tell you a bit about my childhood.
I grew up in Lexington, MA, which is an upper-middle class liberal suburb of Boston. Most of the people I went to school with had parents that either worked at or went to Harvard or M.I.T. – it was a pretty nerdy, intellectual environment. My parents, both computer scientists, moved there for the public schools.
In spite of that, I was a hopeless, pathetic nerd. My idea of fun was practicing classical piano, watching “Amadeus” over and over again, and factoring license plate numbers in my head. When you add to that the facts that I wore glasses, braces, and was chubby, you are talking about one pathetic young nerd girl. When, you top *that* off with the fact that I went through puberty at the wrong time, you can imagine that I went through junior high wondering what everyone was smoking. Oh, and did I mention that my mom hated shopping so I was always wearing one of two bright pink stretch polyester pants? And that my personal hygiene skills were undeveloped? You get the picture.
I was lucky enough to have a best friend starting in 7th grade, who saved me from many pits of despair (although not all). But come high school, my self-esteem was pretty crappy, and the only thing I seemed to be good at, my refuge, was piano and math team.
My parents did an excellent job of not really caring about what I did for the most part, so I wasn’t at all pressured into doing math, and definitely not pressured into doing music. When I came home with an advertisement from a math camp at Hampshire College in western Massachusetts, though, my parents essentially bribed me to go. It didn’t take much convincing, I was intrigued.
Here’s where we get to the title of the post. When I got there, I quickly noticed there were 50 boys and 10 girls. And then I noticed that a bunch of these guys were kind of… cool, they were mostly from places like Stuyvesant and Bronx Science and Evanston, places I’d never heard of but which obviously placed a premium on being a math nerd. Then, this was the miracle, I noticed that these cool, sexy guys, thought I was cool and sexy. OMG, I was a math babe!
It was the first moment I had ever felt like I belonged somewhere, that I was with my peeps. I learned lots of math that first summer, and although most of the specifics kind of wore away over the following year, the feeling that I had a community never did. Actually the one thing I did really learn for good that summer was how to solve the Rubik’s cube using group theory (a subject for another post!). And I distinctly remember carrying around a Rubik’s cube like a piece of platinum my entire junior year of high school, just because it reminded me that I was, in fact, a math babe, at least in one context (although not here! not here whatsoever!).
Which reminds me! This summer, I’m very excited to be going back to the same math camp to teach as a senior staff member. Here’s the list of stuff I have prepared to teach this crop of math studs and math babes:
1) magic squares and generalizations. I just figured out how to generate all 3×3 magic squares! I love those little guys.
2) elementary number theory: fundamental theorem of arithmetic
3) cool geometry stuff like bisectors of angles and sides and all those cool theorems
4) pigeon hole principle, lots of examples
5) euler’s formula and the platonic solids
6) cool stuff with perfect numbers and non-perfect numbers
7) proof by induction, lots of examples
8) basic graph theory
9) bipartite graphs and related theorems.
10) basic ramsey theory
11) more number theory
12) farey fractions
13) continued fractions and the golden ratio
I can’t friggin wait!! Please send me more suggestions if I’m missing something that they really need to know. By the way I’m only teaching the first three weeks, because I couldn’t arrange for the whole 6- the second half they will be learning more specialized subjects from some very cool mathematicians.
Actually there’s another reason I ultimately decided to call this blog “mathbabe,” namely when I googled it, I was first of all offended that the name wasn’t already taken by some other woman math nerd who posting about cool stuff, but what really offended me was that there’s another site with a very similar name which simply shows nearly naked women next to cliff notes on basic math subjects. WTF?!? It is ridiculously obvious to me that math babes should be doing math, not adorning it. So I kind of had to call myself mathbabe after that.
One thing that kind of drives me crazy in economic or business news (which I’m frankly addicted to (which makes me incredibly old and boring)) is the lack of precision exactly when there seems to be some actual data- so at the very moment when you think you’re going to be told what the hard cold facts are, so you can make up your own mind about whether the economy is still sucking or is finally recovering, you get a pseudo-statistic with a side of caveat. I make it a point to try to formally separate the true bullshit from the stuff that actually is pretty informative if you know what they are talking about. I consider “seasonal adjustment” pretty much in the latter category, although there are exceptions (more on that later).
So what does “seasonal adjustment” mean? Let’s take an example: a common one is home sales. It’s a well known fact that people don’t buy as many homes in January and February as they do in May and June- due to some combination of people sitting in their houses eating ice cream straight from the Ben & Jerry’s container when it’s cold outside and the dirty snow tracks on their immaculate rugs during open houses making people trying to sell their houses enraged. So people delay house-hunting til Spring and they delay house-selling til house-hunting starts (side note: because of this, desperate people getting divorced or being forced to move often have to sell their houses at major discounts, so always do your house-hunting right after a huge blizzard).
Considering the cyclical and predictable nature of home sales, people want to “seasonally adjust” the data so that they can discern a move that is *not* due to the time of the year, in other words they want to detect whether a more macroeconomic issue is affecting home sales, such as a recession or housing glut (or both). It’s a reasonable approach- how does it work exactly?
Say you have a bunch of housing data, maybe 20 years of monthly home sales. You see that every single year the same pattern emerges, more or less. Then you could, for a given year, compute the average sale per month for that year. It’s important to compute this average, as we will see, because one golden rule of adjusting data is that the sum of the adjusted data must equal the original data, otherwise you introduce a problem that’s bigger than the one you’re solving.
Once you have the average sale per month, you figure out (using all 20 years) the typical divergence from the average that you see per month, as a percentage of the average per month that year. So for example, January is the worst month for home sales, and in the 20 years of data you see that on average there are 20% fewer home sales in January than there are on the average month of that year, whereas in June there are typically (in your sample) 15% more sales than in the average month that year. Using this historical data, you come up with numbers for each month (-20% for January, 15% for June, etc.). I can finally say what “seasonally adjusted” means: it is the rate of sales for the average month or for the year given these numbers. So if we saw 80,000 home sales in January, and our number for January is -20%, then we will say we have a seasonally adjusted rate of 100,000 sales per month or 1.2 million sales per year.
Note that this system of adjustment follows the golden rule at least for the historical data; by the end of each calendar year, we have attributed the correct overall number of sales, spread out over the months. However, if we start predicting July sales from what we’ve seen from home sales from January to March, taking into account these adjustments, we will also be tacitly assuming an overall number of sales for the year, and the golden rule will probably not hold. This is just another way to say that we won’t really know how many home sales have occurred in a given year until the year is over, so duh. But it’s not hard to believe that knowing these numbers is pretty useful if you want to make a ballpark estimate of the yearly rate of home sales and it’s only March.
A slightly more sophisticated way of doing this, which doesn’t depend as much on the calendar year, is to use the 20 years of data and a rolling 12 month window (i.e. where we add a month in the front and drop off a month in the back and thus always consider 12 consecutive months at a time) to compute the monthly adjustment for each month relative not to the average for the upcoming year, but rather relative to the average of the 12 past months. This has the advantage of be a causal model, (i.e. a model which only uses data in the past to predict the future- I’ll write a post soon about causal modeling) but has the disadvantage of not following the golden rule, at least in a short amount of time. For example, if housing sales are on a slow slide over months and months, this model will consistently fail to predict how low home sale figures should be.
The biggest problems with seasonally adjusted numbers are, in my opinion, that the model itself is never described- do we use 20 years of historical data? 3 years? Do we use a rolling window or calendar years? Without this kind of information, I’m frankly left wondering if you could frigging show me the raw data and let me decide whether it’s good news or bad news.
A few comments have trickled in from friends (over email) who are quants, and I wanted to add them here.
- First, any predicting is hard and assumes a model, i.e., each year is the same, or each month is the same. In other words, as soon as you are talking about something being surprisingly anything, you are modeling, even when you don’t think you are. Most assumptions go unnoticed in fact. Part of being a good quant is simply being able to list your modeling assumptions.
- As we will see when we discuss quant techniques further, a very important metric of a model is how many independent data points you have going into the model- this informs the calculation of statistical significance, for example. The comment then is that modeling seasonal adjustment as I’ve described above lowers your “number of independent data points” count by a factor of 12, because you are basically using all 12 months of a year to predict the _next year_, so what looked like 12 data points is really becoming only one. However, you could try to fit a smaller (than 12) parameter curve to the seasonal data differences, but then there’s overfit from having chosen the family of curves to be one that looks right. More on questions like this when we explore the concept of fitting model to the data, and in particular on how many different models you try for a given data set.
- The final comment is this: all predictions likely violate the golden rule, but the point is you at least want one that isn’t biased, so in expectation it matches the rule.
One of the first things I’d like to set people straight on is what it takes to be a woman in math. The short answer is, a warrior. The longer answer starts like this. At least in this country, in this culture*, it required near-constant resistance to the niggling feeling that you don’t belong, that you are an outsider, and that you will always be an outsider. It takes the belief in yourself as an abstract thinker, as a scientist, and as a _source_ of wisdom. This is completely counter to how the average woman has been taught to behave: demurely, modestly, quietly. Unleaderly. And the above description refers only to the psychological barriers, not the underlying mathematics.
Considering how difficult the material itself is, it’s not surprising how many women drop out eventually.
To be fair, we are seeing many more women finishing college degrees in mathematics and Ph.D.s in mathematics, and that is frigging awesome. But we are still not seeing that many professors, not in the numbers you might think from the Ph.D. programs. Why is this? I think I can explain this at least in part. When one decides to become a math major, it’s a difficult decision in terms of the surrounding cultural expectations, but there’s very good, very consistent feedback (at least outside of Harvard), namely in the form of homework and test grades from undergrad classes. In other words, it may be a weird decision to be a woman in math, but you can *see* your success whenever your homework comes back with a good grade. It’s proof positive that you are doing ok. To some extent in grad school this feedback loop continues, and with luck you have a good advisor who is encouraging and nurturing. However, once outside of grad school the feedback loop all but vanishes and you are left to decide, *within yourself* whether you are good at what you do. This is when you as a woman (and of course this happens to men too but for whatever reason, maybe just hormones, maybe culture, not as often) question yourself, and then look to the outside world for affirmation, and to be honest that’s a pretty tough moment. Many women leave at that moment.
In some sense I am one of them, because I did leave academics. But I left because I decided I wanted more, so more of a moment of strength than a moment of fear. I got a Ph.D. at Harvard, went to M.I.T. for a post-doc, then became an assistant professor at Barnard College. I got to the point where I was pretty sure I’d be able to get tenure, or in other words to the point that I was sure I deserved tenure, and I looked around and decided, this isn’t the kind of feedback loop I want in my life. I need actual feedback, in real time. I left to be a quant in finance (and since then a data scientist at an internet ad company). I feel very lucky that I could make that decision without fear, and I still consider myself a woman in math, and I still encourage women in math to stay in math or at least stay mathematical.
I think if people understood what women in math need to do in order to just be themselves every day, they would be treated less like anomalies and more like superheroes. It’s a tough thing to do, and they should be respected for it. And they are cool. I mean, what’s cooler than someone who lives as an outsider and has come to terms with that? It’s a strength that not everyone has.
Here’s the thing, I don’t want to end this post on a negative note. In spite of everything I’ve said, being a math babe totally rocks, because math rocks. I hope to convincingly illustrate just how much math rocks in future posts.
* I’ve talked to women outside the US about being mathematicians in their country. One thing that commonly comes up is that in Italy, and to some extent France, it is much more common to see women mathematicians. Why is this? One of my Italian women mathematician friends described it to me like this: in Italy, the academic track to become a mathematician is identical to that of becoming a high school math teacher- indeed the two tracks diverge only after a masters degree. The outcome of this system is that it is not seen as a particularly glamorous or even difficult profession- perhaps similar to that of an engineer. According to her, truly ambitious Italians become politicians, not mathematicians.