I wanted to show how to perform a “women on the board of directors” analysis using Bayesian inference. What this means is that we need to form a “prior” on what we think the distribution of the answer could be, and then we update our prior with the data available. In this case we simplify the question we are trying to answer: given that we see a board with 3 women and 7 men (so 10 total), what is the fraction of women available for the board of directors in the general population? The reason we may want to answer this question is that then we can compare the answer to other available answers, derived other ways (say by looking at the makeup of upper level management) and see if there’s a bias.
In order to illustrate Bayesian techniques, I’ve simplified it further to be a discrete question. So I’ve pretended that there are only 11 answers you could possible have, namely that the fraction of available women (in the population of people qualified to be put on the board of directors) is 0%, 10%, 20%, …, 90%, or 100%.
Moreover, I’ve put the least judgmental prior on the situation, namely that there is an equal chance for any of these 11 possibilities. Thus the prior distribution is uniform:
The next step is to update our prior with the available data. In this case we have the data point that there a board with 3 women and 7 men. In this case we are sure that there are some women and some men available, so the updated probability of there being 0% women or 100% women should both be zero (and we will see that this is true). Moreover, we would expect to see that the most likely fraction will be 30%, and we will see that too. What Bayesian inference gives to us, though, is the relative probabilities of the other possibilities, based on the likelihood that one of them is true given the data. So for example if we are assuming for the moment that 70% of the qualified people are women, what is the likelihood that the board ends up being 3 women and 7 men? We can compute that as (0.70)^3*(0.30)^7. We multiply that by 1/11, the probability that 70% is the right answer (according to our prior) to get the “unscaled posterior distribution”, or the likelihoods of each possibility. Here’s a graph of these numbers when I do it for all 11 possibilities:
In order to make this a probability distribution we need to make sure the total adds up to 1, so we scale to get the actual posterior distribution:
What we observe is, for example, that it’s about twice as likely for 50% of women to be qualified as it is for 10% of women to be qualified, even though those answers are equally distant from the best guess of 30%. This kind of “confidence of error” is what Bayesian inference is good for. Also, keep in mind that if we had had a more informed prior the above graph would look different; for example we could use the above graph as a prior for the next time we come across a board of directors. In fact that’s exactly how this kind of inference is used: iteratively, as we travel forward through time collecting data. We typically want to start out with a prior that is pretty mild (like the uniform distribution above) so that we aren’t skewing the end results too much, and let the data speak for itself. In fact priors are typically of the form, “things should vary smoothly”; more on what that could possibly mean in a later post.
Here’s the python code I wrote to make these graphs:
Here’s the R code that Daniel Krasner wrote for these graphs:
I was looking through an old photo album (the kind where there are sticky pages and actual physical photos- it looks like an ancient technology now) and I came across one of my favorites of all time- a picture of me being embraced and supported by Cora Sadosky on one side and Barry Mazur on the other. This picture was taken in 1993 in Vancouver, where I received the Alice T. Schafer prize. It was a critical moment for me, and both of those people have influenced me profoundly. Barry became my thesis advisor; part of the reason I went into number theory was to become his student (the other part was this book).
Cora became my mathematical role model and spiritual mother. I already wrote earlier about how going to math camp when I was 14 changed my life and made me realize there is a whole community of math nerds out there and that I belonged to that nerd community. Well, Cora, whom I met when I was 21, was the person that made me realize there is a community of women mathematicians, and that I was also welcome to that world.
Actually it was something I didn’t even really want to know at the time. After all, I was happy to be a successful math undergraduate at UC Berkeley, frolicking in the graduate student lounge and partaking in tea every day at 3:00. Who cares that I was a woman? It seemed antiquated to me, almost crude, to mention my gender. When I got word that I’d won the prize, my reaction was essentially, “is there money?” (there was a bit).
And when I meet young women in math nowadays with that attitude, I am happy for them, really very happy for them. To live in that state of not caring what your gender is in mathematics is a kind of bliss, that lasts until the very moment it stops. My greatest wish for future generations of women in math is for that bliss to never stop.
And yet. I went to Vancouver and met Cora and learned about Alice Shafer and her struggles and successes as a trailblazer for women in math, and I felt really honored to be collecting an award in her name. And I felt honored to have met Cora, whose obvious passion for mathematics was absolutely awe-inspiring. She was the person who first explained to me that, as women mathematicians, we will keep growing, keep writing, and keep getting better at math as we grow older (unlike men who typically do their best work when they’re 29), and we absolutely have to maintain a purpose and a drive and fortitude for that highest call, the struggle of creation.
I kept up with Cora over the years. Every now and then she’d write to me and send me pushy little maternal notes reminding me to work hard and stay strong and productive. And I’d write to her with news of my life and my growing family and sometimes when I visited D.C. I’d meet her and we’d have lunch or dinner and talk about ideas and great books we’d read and how much we loved each other.
When I googled her this morning, I found out she’d died about 6 months ago. You can read about her difficult and inspiring mathematical career in this biography. It made me cry and made me think about how much the world needs role models like Cora.
First of all, I changed the theme of the blog, because I am getting really excellent comments from people but I thought it was too difficult to read the comments and to leave comments with the old theme. This way you can just click on the word “Go to comments” or “Leave a comment” which is a bit more self-evident to design-ignorant people like me. Hope you like it.
Next, I had a bad day today, but I’m very happy to report that something has raised my spirits. Namely, Jake Porway from Data Without Borders and I have been corresponding, and I’ve offered to talk to prospective NGO’s about data, what they should be collecting depending on what kind of studies they want to be able to perform, and how to store and revise data. It looks like it’s really going to happen!
In fact his exact words were: I will definitely reach out to you when we’re talking to NPOs / NGOs.
Oh, and by the way, he also says I can blog about our conversations together as well as my future conversations with those NGO’s (as long as they’re cool with it), which will be super interesting.
Oh, yeah. Can I get a WOOHOO?!?
In a previous post, I wrote about what I see as the cowardice and small-mindedness of the U.S. government and in particular the regulators for not demanding daily portfolios of all large investors. Of course this goes for the governments in Europe as well, and especially right now. The Economist had a good article this past Friday which attempted to quantify the results of a Greek default, but there were major holes, especially in the realm of “who owns the CDS contracts on Greek bonds, and how many are there?”. This fear of the unknown is a root cause of the current political wrangling which will probably end in a postponement of resolving the Greek situation; the question is whether the borrowed time will be used properly or squandered.
It’s ridiculous that nobody knows where the risk lies, but as a friend of mine pointed out to me last week at lunch, it probably won’t be enough to demand the portfolios daily, even if you had the perfect quantitative risk model available to you to plug them into. Why? Because if “transparency” is what the regulators demand, then “transparency” is what they would get – in the form of obfuscated lawyered-up holding lists.
In other words, let’s say a bank has a huge pile of mortgage-backed securities of dubious value on their books, but doesn’t want to accept losses on them. If they knew they’d have to start giving their portfolio to the SEC daily instead of quarterly, it would change the rules of the game. They’d have to hide these holdings by pure obfuscation rather than short-term month- or quarter-end legal finagling. So for example, they could invest in company A, which invests in company B, which happens to have a bunch of mortgage-backed securities of dubious value, but which is too small to fall under the “daily reporting” rules. This is just an example but is probably an accurate portrayal of the kind of thing that would happen with enough lead time and enough lawyers.
What we actually want is to set up a system whereby banks and hedge funds are motivated to be transparent. Read this as: will lose money if they aren’t transparent, because that’s the only motivation that they respond to.
In some sense, as my friend reminded me, we don’t need to worry about hedge funds as much as about banks. This is because hedge funds do their trades through brokerages, which force margin calls on trades that they deem risky. In other words, they pay for their risk through margins on a trade-by-trade, daily basis. If you are thinking, “wait, what about LCTM? Isn’t that a hedge fund that got away with murder and almost blew up the system and didn’t seem to have large margins in place?” then the answer is, “yeah but brokers don’t get fooled (as much) by hedge funds anymore”. In other words, brokers, who are major players in the financial game, are the policemen of hedge funds.
There are two major limits to the above argument. Firstly, hedge funds purposefully use multiple brokers simultaneously so that nobody knows their entire book, so to the extent that risk of portfolio isn’t additive (it isn’t), this policing method isn’t complete. Secondly, it is only a local kind of risk issue- it doesn’t clarify risk given a catastrophic event (like a Greek default), but rather a more work-a-day “normal circumstances” market risk.
Even so, what about the banks? Are there any brokers measuring the risk of their activities and investments? Since the banks are the brokers, we have to look elsewhere… I guess that would have to be at the government, and the regulators themselves, maybe the FDIC… in any case, people decidedly not players in the financial game, not motivated by pay-off, and therefore not prone to delving into the asperger-inspiring details of complicated structured products to search out lies or liberal estimates.
The goal then is to create a new kind of market which allows insiders to bet on the validity of banks’ portfolios. You may be saying, “hey isn’t that just the stock price of the bank itself?”, and to answer that I’d refer you to this article which does a good job explaining how little information and power is actually being exercised by stockholders.
I will follow up this post with another more technical one where I will attempt to describe the new market and how it could (possibly, hopefully) function to motivate transparency of banks. But in the meantime, feel free to make suggestions!
This is a co-post with FogOfWar.
Here’s an interesting article about how many board of directors for S&P500 companies consist entirely of men. Turns out it’s 47. Well, but we’d expect there to be some number of boards (out of 500) which consist entirely of men even if half of the overall set of board members are women. So the natural question arises, what is the most likely actual proportion of women given this number 47 out of 500?
In fact we know that many people are on multiple boards but for the sake of this discussion let’s assume that there’s a line of board seekers standing outside waiting to get in, and that we will randomly distribute them to boards as they walk inside, and we are wondering how many of them are women given that we end up with 47 all-men boards out of 500. Also, let’s assume there are 8 slots per board, which is of course a guess but we can see how robust that guess is by changing it at the end.
By the way, I can think of two arguments as to why the simplification that nobody is on multiple boards argument might skew the results. On the one hand, we all know it’s an old boys network so there are a bunch of connections that a few select men enjoy which puts them on a bunch of boards, which probably means the average number of boards that a man is on, who is on at least one board, is pretty large. On the other hand, it’s also well-known that, in order to seem like you’re diverse and modern, companies are trying to get at least a token woman on their board, and for some reason consider the task of finding a qualified woman really difficult. Thus I imagine it’s quite likely that once a woman has been invited to be on a board, and she’s magically dubbed “qualified,” then approximately 200 other boards will immediately invite that same woman to be on their board (“Oh my god, they’ve actually found a qualified woman!”). In other words I imagine that the average number of boards a given woman is on, assuming she’s on one, is probably even higher than for men, so our simplifying assumptions will in the end be overestimating the number of women on boards. But this is just a guess.
Now that I’ve written that argument down, I realize another reason our calculation below will be overestimating women is this concept of tokenism- once a board has one woman they may think their job is done, so to speak, in the diversity department. I’m wishing I could really get my hands on the sizes and composition of each board and see how many of them have exactly one woman (and compare that to what you’d expect with random placement). This could potentially prove (in the sense of providing statistically significant evidence for) a culture of tokenism. If anyone reading this knows how to get their hands on that data, please write!
Now to the calculation. Assuming, once more, that each board member is on exactly one board and that there are 8 people (randomly distributed) per board, what is the most likely percentage of overall women given that we are seeing 47 all-male boards out of 500? This boils down to a biased coin problem (with the two sides labeld “F” and “M” for female and male) where we are looking for the bias. For each board we flip the coin 8 times and see how many “F”s we get and how many “M”s we get and that gives us our board.
First, what would the expected number of all-male boards be if the coin is unbiased? Since expectation is additive and we are modeling the boards as independent, we just need to figure out the probability that one board is all-male and multiply by 500. But for an unbiased coin that boils down to (1/2)^8 = 0.39%, so after multiplying by 500 we get 1.95, in other words we’d expect 2 all-male boards. So the numbers are definitely telling us that we should not be expecting 50% women. What is the most likely number of women then? In this case we work backwards: we know the answer is 47, so divide that by 500 to get 0.094, and now find the probability p of the biased coin landing on F so that all-maleness has probability 0.094. This is another way of saying that (1-p)^8 = 0.094, or that 1-p is 0.744, the eighth root of 0.094. So our best guess is p = 25.6%. Here’s a table with other numbers depending on the assumed size of the boards:
If anyone reading this has a good sense of the distribution of the size of boards for the S&P500, please write or comment, so I can improve our estimates.
A nerd friend of mine kindly rewrote my python scripts in R and produced similar looking graphs. I downloaded R from here and one thing that’s cool is that once it’s installed, if you open an R source code (ending with “.R”), an R console pops up automatically and you can just start working. Here’s the code:
gdata <- read.csv('large_data_glucose.csv', header=TRUE)
#We can open a spreadsheet type editor to check out and edit the data: edit(gdata) #Since we are interested in the glucose sensor data, column 31, but the name is a bit awkward to deal with, a good thing to do is to change it: colnames(gdata) <- "GSensor" #Lets plot the glucose sensor data: plot(gdata$GSensor, col="darkblue") #Here's a histogram plot: hist(gdata$GSensor, breaks=100, col="darkblue")
#and now lets plot the logarithm of the data: hist(log(gdata$GSensor), breaks=100, col="darkblue")
And here are the plots:
One thing my friend mentions is that R automatically skips missing values (whereas we had to deal with them directly in python). He also mentions that other things can be done in this situation, and to learn more we should check out this site.
R seems to be really good at this kind of thing, that is to say doing the first thing you can think about with data. I am wondering how it compares to python when you have to really start cleaning and processing the data before plotting. We shall see!
I want to describe the culture of working at D.E. Shaw during the credit crisis, so from June 2007 to June 2009, because I think it’s emblematic of something that most news articles and books written about hedge funds really miss out on when they fixate on the average I.Q. of the people working there, which is in the end a distraction and nothing more, or the bizarre or quirky personalities that exist there, which is only idiosyncratic and doesn’t explain anything deeply.
I promised myself I’d put focus on the following phrase, which struck me down when I first heard it used and still makes me shake my head, namely the concept of “dumb money.” The phrase was tossed around constantly and cleverly, and really, to understand what it means inside the context of the hedge fund culture, is to understand the culture. So I’ll try to explain it. First a bit of context.
Most of the quants at D.E. Shaw were immigrant men. In fact I was the only woman quant when I joined, and there were quite a few quants, maybe 50, and I was also one of the only Americans. What nearly all these men had in common was a kind of constant, nervous hunger, almost like a daily fear that they wouldn’t have enough to eat. At first I thought of them as having a serious chip on their shoulder, like they were the kind of guy that didn’t make the football team in high school and were still trying to get over that. And I still think there’s an element of something as simple as that, but it goes deeper. One of my colleagues from Eastern Europe said to me once, “Cathy, my grandparents were coal miners. I don’t want my kids to be coal miners. I don’t want my grandchildren to be coal miners. I don’t want anybody in my family to ever be a coal miner again.” So, what, you’re going to amass enough money so that no descendent of yours ever needs to get a job? Something like that.
But here’s the thing, that fear was real to him. It was that earnest, heartfelt anxiety that convinced me that I was really different from these guys. The difference was that, firstly, they were acting as if a famine was imminent, and they’d need to scrounge up food or starve to death, and secondly, that only their nuclear family was worth saving. This is where I really lost them. I mean, I get the idea of acts of desperation to survive, but I don’t get how you choose who to save and who to let die. However, it was this kind of us-against-them mentality that prevailed and informed the approach to making money.
Once you understand the mentality, it’s easier to understand the “dumb money” phrase. It simply means, we are smarter than those idiots, let’s use our intelligence to anticipate dumb peoples’ trades and take their money. It is our right as intelligent, imminently starving people to do this. Chasing dumb money can take various forms, but is generally aimed at anticipating lazy fund managers: if you know that they always wait until Friday afternoon to balance their books, or that they wait until the end of the month, or that they are required to buy certain kinds of things, you can anticipate their trades, make them yourself a bit before they do, thereby forcing them to pay more, and getting a nice little profit for yourself. In short this works in general, since statistically speaking the anticipated trade wasn’t driving up the intrinsic value of the underlying, but rather was being affected by trade impact for a short amount of time. If we can anticipate big trades by lots of dumb money, then the short-term market impact will be large enough and last long enough to buy in beforehand and sell at the top, while it still lasts, assuming there’s sufficient liquidity. The subtext of taking dumb money, going back to the football team issue, is: if we don’t somebody else will, and then we will feel like fools for not doing it ourselves.
To tell you the truth, I was completely naive when I went to work there. I had kind of accepted the job because I wanted to be a business woman, wanted a brisk pace after the agonizing slowness of academics, and I had really no moral judgment on the concept of a hedge fund; I thought it was morally neutral, at worst a scavenger on the financial system, like a market maker or someone who provides insurance for something. Well I’ve decided it’s more like a leech.
Getting to the part about actually working with Larry Summers. I did work on a couple of his ideas, although in order not to get sued I can’t be detailed about what his ideas were. And I had various meetings with him and a bunch of managing directors. One thing I remember about these meetings was the eery way the managing directors seemed intimidated by him, even though behind his back they kind of scoffed at the possibility that he could actually offer good modeling ideas. It was basically a publicity stunt, or at least rumored to be, to have him work there. It was after he had gotten pushed out of the Presidency at Harvard for talking out of his ass about women in math, and yes it was a bit surreal to be the only woman quant in the place, and to be working on his project considering that. Since I am pretty much never intimidated for some reason, I had no problem. He kept on grilling me about various things to try and I kept explaining what I’d done and how I’d already thought of that. It was fine, pretty combative and pushy, but actually kind of fun. I really have nothing to say about him treating me differently because I was a woman.
But when I think about that last project I was working on, I still get kind of sick to my stomach. It was essentially, and I need to be vague here, a way of collecting dumb money from pension funds. There’s no real way to make that moral, or even morally neutral. There’s no way to see that as scavenging on the marketplace. Nope, that’s just plain chasing after dumb money, and I needed to quit. I still don’t know if that model went into production.