There should be a macho way to say “I don’t know”
I recently gave an interview with Russ Roberts at EconTalk, which was fun and which has generated a lot of interesting feedback for me. I had no idea so many people listened to that podcast. Turns out it’ll eventually add up to something like 50,000, with half of those people listening this week. Cool!
One thing Russ and I talked about is still on my mind. Namely, how many problems are the direct result of people pretending to understand something, or exaggerating the certainty of an uncertain quantity. People just don’t acknowledge errorbars when they should!
What up, people?
Part of the problem exists because when we model something, the model typically just comes out with a single answer, usually a number, and it seems so certain to us, so tangible, even when we know that slightly different starting conditions or inputs to our models would have resulted in a different number.
So for example, an SAT score. We know that, on a different day with a different amount of sleep or a different test, we might score significantly differently. And yet the score is the score, and it’s hugely important and we brand ourselves with it as if it’s some kind of final word.
But another part of this problem is that people are seldom incentivized to admit they don’t know something. Indeed the ones we hear from the most are professional opinion-holders, and they are going to lose their audience and their gigs if they go on air saying, “I’m not sure what’s going to happen with [the economy], we’ve honestly never been in this situation before and our data is just not sufficient to make a prediction that’s worth its weight.”
You can replace “the economy” by anything and the problem still holds.
Who’s going to say that?? Someone who doesn’t mind losing their job is who. Which is too bad, because honest people do say that quite a large portion of the time. So professional opinion-holders are kind of trained to be dishonest in this way.
And so are TED talks, but that’s a vent for another day.
I wish there were a macho way to admit you didn’t know something, so people could understand that admitting uncertainty isn’t equivalent to being wishy-washy.
I mean, sometimes I want to bust out and say, “I don’t know that, and neither do you, motherfucker!” but I’m not sure how well that would go over. Some people get touchy about profanity.
But it’s getting there, and it points to something ironic about this uncertainty-as-wishy-washiness: it is sometimes macho to point out that other people are blowing smoke. In other words, I can be a whistle blower on other people’s illusion of certainty even when I can’t make being uncertain sound cool.
I think that explains, to some extent, why so many people end up criticizing other people for false claims rather than making a stance on uncertainty themselves. The other reason of course is that it’s easier to blow holes in other people’s theories, once stated, than it is to come up with a foolproof theory of one’s own.
Any suggestions for macho approaches to errorbars?
mathbabe 
“What a stupid question.”
LikeLike
But what if it’s not a stupid question?
LikeLike
I thought we were trying to be macho!
LikeLike
I’ll be your straight man anytime!
LikeLike
Three words that are hard to say… “I love you.” and “I don’t know.” The first three get the girl. Mathbabe needs to work on her gender peers to do the same for the second three.
I’m here because of your Econtalk appearance.
LikeLike
Welcome! What’s a gender peer?
LikeLike
A phrase I just made up. Other women.
LikeLike
You’re right. It’s a big problem. I wrote a post on it in 2010 – http://humanists4science.blogspot.co.uk/2010/07/down-with-naked-numbers.html.
It’s not just numbers. When Tony Blair told Parliament that we should invade Iraq because there was strong evidence that Iraq had WMDs it wasn’t the claim about WMDs that was wrong so much as the claim that the evidence was strong. That was just a lie. But no media person focused on that so all we got was conflicting opinions (and, obviously, an immoral war).
LikeLike
Actually, I’m usually rather impressed when people say “I don’t know”, especially if they elaborate upon how much they don’t know and inform me in an up-front way. It seems macho enough to me.
LikeLike
+1 to Anonymous Coward. I find that starting out by communicating what I don’t know (and why) can disarm people waiting to poke holes in my thinking. I’m so confident I can point out my weaknesses first. Poke a hole in that!
LikeLike
“I don’t know. Right now, we *can’t* know. In fact, if anyone actually knew, they’d be doing something about it instead of talking about it.”
LikeLike
You may be looking at this the wrong way. I’m of the opinion that certitude IS macho, and the problem is the desire for machismo. The solution, then, may be to undermine the perception of machismo as a desirable trait.
By the way, there is a whole movement whose raison d’être is to encourage people to entertain doubt in their beliefs. It’s called Skepticism.
LikeLike
Yes, what I was thinking is that trying to be “macho” in the first place is to try to lead a logical discussion with an illogical appeal to emotion. It just doesn’t fit. But it’s amazing how there’s an entire circuit of “intellectuals” based on what is a very old and well-known fallacy.
LikeLike
By learning to admit: The more I know the more I know that I don’t know. Or, as Albert Einstein said: “As our circle of knowledge expands, so does the circumference of darkness surrounding it.” I like to reframe that to: Our ignorance increases with the circumference of our knowledge. – Long live the paradox, because they challenge our entrenched thinking!
LikeLike
Last month I gave a presentation to a potential client about how I would approach a security issue they had within their IT environment. As a consultant, I’m obviously trying to actually pitch what effect my services have, not necessarily how I achieve the goal. This time was a bit different, there were 14 people in the room, some being aggressive with me. At one point someone asked (paraphrase) How does xyz work then? I paused for 3.5 seconds and said I don’t know. The person asking smirked. Immediately the head of the group said that’s the best response I’ve heard all day.
LikeLike
Chris, you are my hero! -Cathy
LikeLike
“I bet my error bars are bigger than yours.”
LikeLike
Thread winner!
LikeLike
+ a really big 1!
LikeLike
Like this one too…!
LikeLike
It may be that hubris outweighs humility for Darwinian, historical – almost genetic reasons. Maybe a bit of humility is a new ethical paradigm which is not (yet) wired in. There are human limits to our ability to process, and that hubris is only going to get worse. I read just today on MIT blog that
“human senses have a limited bandwidth–our brains can receive information from the external world at roughly gigabit rates. So a computer simulation at exascale data rates simply overwhelms us. The famous aphorism compares data overload to drinking from a fire hose. This is more like stopping a tidal wave with a bucket.”
quote from http://www.technologyreview.com/view/510371/how-bullet-time-will-revolutionise-exascale-computing/?utm_campaign=newsletters&utm_source=newsletter-weekly-computing&utm_medium=email&utm_content=20130214
Computing makes the i don’t know into I can’t possibly know. We will have to wait for serious upscaling of our brainpower! Along with the necessary dosage of augmented humility!
I love to hear your take on this problem Cathy.
LikeLike
I agree this is a big problem. Definitely in finance.
An important part of my interview process was seeing how well people calibrated their uncertainty. That is, could they assess how far off their estimates could be. People in general are very bad at that.
But, business pressures and cultural biases are also part of it. I was generally kept away from clients because of my tendency to refer to forecasts as “educated guesses” and things like that.
The best I’ve come up with on the machismo regard is “I don’t know and neither does anyone else”. But Keith has a good point that trying to be macho is part of the problem, not the solution.
LikeLike
I’ve been working on that problem with our sales guy and our CEO just recently. CEO and sales guy don’t want any uncertain words in presentations because they scare people. I think leaving all the uncertainty out of the presentation leads to customers who expect magic (and turns off the ones who know better). The latter category is a customer I want. The former–they’re usually too expensive to keep.
Honestly: I don’t know how to solve this.
LikeLike
Exactly. In some contexts at least, it’s not machismo. It’s playing to the risk aversion of the client. And the client is always right. Or else they’re not your client any more.
As you note, there is no good way to solve this one. It’s on the customer side, not the business side.
LikeLike
It reminds me of what William Easterly (BTW, I first heard him from Econtalk about development economics) said “I don’t know” is an extremely unpopular answer, among just everybody. He then continued on to say as a joke “I’ve had a lot of difficulty raising funds for the ‘I don’t know’ Foundation”.
Source: http://www.carnegiecouncil.org/studio/multimedia/20110915/index.html
LikeLike
You’re tackling the core problem of western thought here. Socrates, as channeled through Plato, died for asking this kind of question. He “corrupted the young” by casting serious doubt our ability to know anything. Doggedly pursuing this angle led to a cup of hemlock. Society just couldn’t tolerate him.
Plato went on to construct a pretty cogent philosophy out of the contentious groundwork that his teacher laid and this has held some sway for 2500 years, but fool-proof? No. The impulse to claim knowledge as opposed to admitting uncertainty is pretty universal. Platonism became Catholicism. Religions, ideologies and, hey, even cosmetic products are born and thrive in the face of so much contrary evidence.
Socrates was right, as you note, but whether we’re selling lip gloss or junk bonds or ourselves as the best candidate for a job, the majority of our fellow humans will glaze over or just turn away if we don’t express some degree of certainty. Humanity loves to be conned.
LikeLike
We-e-e-ell… he mostly died for having been a mentor to some of the Thirty Tyrants. The Socrates of the Trial is a theatrical character.
LikeLike
I’d suggest playing devil’s avocate with your pet theory or pet explanation, the one you take as “more likely than not”. So, you have your “pet explanation” T, and at first you argue in favor of T, and then in a second part you argue against T (the devil’s advocate part). I think one can become emotionally or ego-involved with a “pet theory”, so by playing devil’s advocate one can cultivate a skeptical approach, forcing one to present the case against one’s pet theory …
And I think Cathy raises an important question in this post.
LikeLike
I’m here also due to your conversation with Russ Roberts which was very entertaining. Anyway, I recently read a book about the 16th century french philosopher Michel de Montaigne who made an absolute virtue of uncertainty. It gave me a feeling of strength to think it’s good to be in the openminded state of wonder about all things.
LikeLike
Beats me.
LikeLike
In mathematics, I think the “macho” way to say “I don’t know” is to say, “Nobody knows the answer to that.” Even more dramatic is to say, “That’s the [BLANK] conjecture, and nobody knows how to solve that.” However, I usually prefer to say a few words about why I don’t know the answer, or where I could look to start to find the answer, e.g., “Lewis has a book on the Hodge conjecture.” On the other hand, I rarely have to deal with jerks who sneer at me just because I don’t know the answer to a math question.
LikeLike
“There isn’t an exact answer, and anyone who says they have one is lying”?
LikeLike
Thanks for raising this important topic. I find that the older I get, the less I know about all manner of things. Life is humbling, ultimately. Personally, I don’t feel the need to put a macho spin on humility.
So how do we live with acute awareness of the limits of our knowledge? I don’t know. As my favorite poet Mary Oliver says, “There are things you can’t reach. But you can reach out to them.” It’s the reaching that matters, not the grasping.
LikeLike
Cathy O’Neal, you might be interested in a paper and series of posts by Judith Curry at Climate Etc. Titled “The Uncertainty Monster,” they deal with … uncertainty, primarily with climate models.
http://judithcurry.com/?s=the+uncertainty+monster
LikeLike
I’ll show you my beta if you show me your covariance…?
LikeLike
Excellent podcast with Russ. I listen to them every week. I thought you both agreed more than you disagreed. You both were uncertain on the correct response but I thought you both admitted you did not know.
Being an internet person who has started and run businesses sometimes it is best to try something and tweak it as you go along. The trouble with that approach is that “legal” rules look for certainty and there is often no revision. As I like to say, it is the lawyers job to eliminate risk and the business persons job to manage risk.
With so many lawyers in Congress who think passing a law is solving a problem I feat that we will not generate a real solution. What we actually need is action not new laws. There are plenty of statutes on fraud on the books yet there has been no action prosecuting people on Wall Street that we bailed out.
I hope for a solution but fear that Lawrence Lessig is correct that the system is now corrupt. There are too many monied interests involved that prevent congress from acting. A rant of my own for another day.
LikeLike
“There are known knowns; there are things we know we know.
We also know there are known unknowns; that is to say, we know there are some things we do not know.
But there are also unknown unknowns – the ones we don’t know we don’t know.”
What, Rumsfeld isn’t macho?
LikeLike
Actually, “unknown unknowns” goes back to Daniel Ellsberg. As Rumsfeld and Ellsberg were both in the defense establishment in the 60s, they probably ran across each other.
Rumsfeld’s problem was his reaction to unknown unknowns was Rhett Butler’s…
“I don’t give a damn”
LikeLike
Loved the interview—big fan of mathbabe.org and Econ Talk.
One person who seems to embody a sense of machismo in his uncertainty is Nassim Taleb. He’s accused of being an arrogant jerk (not without reason), but I think he’s just so tired of people screwing up the system out of supposed “certainty” that he realizes being brash is the only way to go. I’m starting to see that his prickly, uber-confident style might be the only way to talk about uncertainty if you want to get people to listen.
LikeLike
Hi Cathy:
I had heard about EconTalk from several sources, and this was the first episode I listened to. You were fantastic. I’d love it if you started a podcast of your own.
I think I tend to hear “I don’t know” more often in the hard sciences rather than in industries where uncertainty is considered a weakness. And I agree with the commenters who say that the most likely successful path is tying “I don’t know” with “and neither does anyone else.”
I agree with Anonymous Coward, way up at #9: I love to hear an explanation about why an expert doesn’t know, especially when they elaborate on what they do and don’t know. It both makes for an interesting mystery and lends credence to their testimony. And it sometimes even points out directions for how we might eventually know.
If you do it right, you can come off much more trustworthy with “I don’t know.”
LikeLike
Thanks for your kind words, Dwayne.
Cathy
LikeLike
Causal arguments are always inductive, not deductive. Data cannot guarantee anything about the future. While data can provide strength to an inductive argument, nobody can guarantee anything about the future. Thus, none of us can know anything about the future. We simply can believe things will happen with varying degrees of certainty.
LikeLike
But you are convinced that the model says global warming is real.
LikeLike
Certainly the expert I talked to managed to explain all the arguments against it that I’ve heard as well as describing the model that he uses that convince him it’s real. I don’t claim to understand it minutely.
LikeLike
Recognizing uncertainty doesn’t mean you can’t also recognize that the balance of the probability distribution is on one side of the issue.
Though, I get frustrated with the climate debate because even if it is not a problem, ocean acidification would be a sufficient reason to worry about CO2 in the atmosphere.
LikeLike
Uh, no. No payoff to “I don’t know”. People like to hear what they want to hear, lies or no. I’m positioning to alpha if I tell you what you want to hear first. That’s the way I get elected 🙂
LikeLike
I’ve discovered your blog thanks to the Econtalk podcast, and I really love it! I actually agree with your set of solutions more than his. Regulation needs to be there, be clear, be strong and predictable with good and fair enforcement. Everything the SEC is not.
On the uncertainty side, there is one person I can think of immediately who could say “I don’t know” a hundred different ways and still come off as a badass: Socrates.
LikeLike
I am a regular listener of Russ and I should thank him introducing me to your work. I am closer to you than him in my leanings and have to say you did a fantastic job in his show. Keep up the good work!
LikeLike
I got here from Econtalk as well. You said “how many problems are the direct result of people pretending to understand something, or exaggerating the certainty of an uncertain quantity.” I think this is another huge benefit of both Nate Silver and Russ Roberts in that they dig into why pundits and researchers are incentivized to promulgate incorrect information. What good is machismo if you are out of a job?
LikeLike
Mathbabe,
Great discussion with Russ Roberts. Am a fan of both your work. I’m not sure if I completely agree with all of your conclusions. In particular, you talked down hedge funds for their arrogance based on your experience at DE Shaw and RiskMetrics.
I work for a hedge fund software provider, including a previous stint in a customer facing role. Believe me, they ask about a lot more than just formatting. If the numbers are unexpected or not understood, we hear about it quickly and frequently. The reason I like hedge funds, as compared to investment banks, is that they are typically managing their own money alongside that of clients. This is aligned with what the investment banks when they were still private partnerships, as opposed to being intentionally Too Big To Fail (TBTF). Arrogance is an unintended “bug” of a capitalistic system, particularly for the successful, but as you saw, it is exposed as a weakness. The arrogant become blind to their soft spots. If this happens to relatively small players, the system puts them back in their place.
Once the players become TBTF, the dynamic changes. They gain influence over politics, not necessarily having to institute sensible risk management. What’s the role of risk management and modelling, when a number of these blowups like the London whale at JP Morgan are caused by copy and paste errors in Excel? In my opinion, the best possible solution is the same one used when dealing with spaghetti code: decouple the big balls of spaghetti. Pull them apart. This way they stop having negative social side effects (externalities in econspeak) if something goes wrong. Once banks stop being too big to fail, the whole dynamic will change. I may be wrong, or maybe not. I don’t know. 🙂
LikeLike
Agreed. But I hope I made it clear that it was the too-big-to-fail banks that cared about formatting over risk. The quantitative hedge funds were definitely the smartest and most careful of customers.
LikeLike
I have to give a plus 1 to the Taleb suggestion. He is very aggressive with his, I don’t know. In short his philosophy is old school classical skepticism. You cant predict what will happen but you can prepare yourself and say FU to fate. You are probably aware of him but if you aren’t, he is an ally when it comes to the big banks.
LikeLike
i agree with all of this and my joke answer, going off of your profanity point, is to get samuel l jackson to make a viral video on the subject.
on the other hand, i also feel like this is kind of like asking Why can’t we just be non-violent? or, really, Why can’t we just get rid of competition? because if you are in a setting of competition and you are willing to admit holes in your knowledge, someone else is going to fill in those holes.
(now i’m imagining someone on the news-tee-vee being asked about the economy and replying “How the hell should i know?!” and if it’s not a stupid question, the follow-up would be “That’s a great question, but anyone who says they know the answer is lying!” but actually, this reminds me, i’d just rather everyone be less macho.)
LikeLike
+1 to those who enjoyed your conversation with Russ Roberts. I really appreciated his clear expression that he’s at least as disgusted with the current regulatory system as you are, and you pausing and seeming to be able to take that in. I like that you floated the response that you were more concerned about the little guy, and Russ was pretty firm that, no, in fact, as a capitalist who embraced the importance of profit and loss, he didn’t think you were more concerned than he was about the little guy’s well-being.
A very rare civil exchange between sophisticated advocates of two at-odds worldviews. We’re so hungry for that I found my eyes moist as I listened.
LikeLike
Thank you!
LikeLike
I listen to Econtalk all the time and this episode was a great one. I really enjoyed listening to you.
Three things:
1. Being able to say “I don’t know” sounds like what Nassim Taleb (Black Swan) would say, who is disgusted by those who think they can accurately predict the future and thus don’t think enough about “what could happen if we are wrong?” I’m surprised Russ did not mention Taleb during the discussion.
2. I think where you and Russ have a lot in common is that you both appreciate “clean”. You like that math is “clean” and he would like to see perfectly “clean” capitalism.
3. On morality in the banking and finance sector: that will never happen. All that matters is force. In this case force = votes. Vote for politicians who will force politics out of banking and finance with law and regulation.
LikeLike
There is also sometimes no feminine way to say “I don’t know.”
e.g. in response to “Who’s your child’s father?”
Always better to respond “my husband, of course!”
LikeLike
Assertive conversation can be an eyeopener, or a door closer, depending on who you are talking to/with. The most difficult problem is to lay out all the facts, and then work on the unknowns around the facts and settle on either new facts that complete the “conversation”, or a list of unknowns to recognize as impediments to “ending” the conversation constructively.
If we are talking long distance, on the phone perhaps, about the economy, and I quip, “now the sky is blue”, and you are looking at a cloudy sky, would you provide an unqualified, “no it’s not”, or would you say “not where I am at”, or would you just ignore the “weather facts”, and move on, because all those facts are not pertinent to what you believe the conversation is centered on.
Factualists (I made that up) are everywhere. The whole conversation about the state of our economy, between the political parties in the U.S., is centered on unimportant facts, such as “unemployment is still over 5%”, or “he hasn’t fixed the economy as he promised”. All the other “facts” or “i don’t knows” are left in the dust in those conversations, because they don’t allow the same point to be made.
Ultimately this is all about agenda… If you don’t have one, you probably are not burdened by the feelings that keep you from saying “I don’t know”.
LikeLike
Unrelated: I am LONG time Econtalk listener and your episode was one of my favorites. What I liked best is that you and Russ are not idealogically equal, but your overlaps are very interesting (someone should draw that ven diagram). I think it is where a lot of people live. I would NEVER have thought of myself as someone that could find common ground with an “Occupy-er”, but it just goes to show that intelligent and civil discussion can be enlightening. I guess I am admitting that “I don’t know” everything that I thought I knew.
LikeLike
Personally, I am not surprised to find a lot of common ground. Anyone with an interest in a fair system would be upset with what we have now. I would like to find common ground on which to work to change things. @Stilllearning (and Russ Roberts, if you are listening, or anyone else sympathetic to his views): what do you think should be done to fix the system?
Is it just a matter of “say no to bailouts” or are there active measures you would advocate?
LikeLike
I do not know for sure if I love this discussion… 🙂
LikeLike
I randomly downloaded the podcast just today and I looked at your blog the first chance I got. I am in so much agreement with what you were saying. Keep up the good work. Glad to hear you found a place better than DE Shaw and the internet company du jour.
LikeLike
Thanks for coming!
Cathy
LikeLike
Humans like narrative… it’s why we ask questions, but also look for answers (signal) even when there is none to be found (in the noise). That’s why asking the right question is so important, and why asking the wrong one can be incredibly distracting.
The closest I know of to macho uncertainty is focusing on what happens if we’re wrong… or alternately what happens if we make no decision. Sometimes the risk of one radically outweighs the other… global warming comes to mind… if there is even a 1% risk of the forecast scenario it is catastrophic… so why don’t we at least prepare for the tail risk?
Good luck though 🙂
LikeLike
Mark Twain: “I was gratified to be able to answer promptly, and I did. I said I didn’t know.”
LikeLike
Yes excellent thought. It comes back to media, communication, and popularity.
LikeLike
Expand Western Culture’s affection for boolean logic. In Japan (a culture much more concerned with “face” and worrying about being wrong and thus disgraced) there are THREE possible answers to any question: yes, no and “mu” (sp?). Mu simply means that I do not have enough information to answer your question. Perhaps it CANNOT be answered, perhaps I need to research more to answer it, perhaps we need to wait and see what the answer will be, but regardless, when I say “mu” you say “ok” and we move on with no loss of face or hint of failure or need of machismo.
LikeLike
I have been living in Japan for years, and I have never gotten a “mu” answer, despite asking loads of questions. This “mu” business sounds more like Zen folklore than actual social experience. When it comes to questions of fact, Japanese answers pretty much span the same spectrum as American answers. When it comes to answering requests, there seems to be more hesitancy to outright refuse, so it is easier to get replies like “that may be difficult…”
LikeLike
About Ted Talks, you might like this perspective written on Quora:
http://www.quora.com/TED/Why-do-some-people-hate-TED/answer/Micah-Boswell
LikeLike
Interesting, thanks! I don’t think it’s the same reason I hate TED. But there are commonalities.
LikeLike
This was a debate on the question of whether to do genetic engineering on humans.
http://intelligencesquaredus.org/debates/past-debates/item/798-prohibit-genetically-engineered-babies
The side opposed to genetic engineering really a good way to say “We don’t know and neither do you.”
The side in favor of doing genetic engineering (against the proposition) basically said “This will help people. This is the right thing to do. Prohibiting it would be cruel.” They showed no doubt either about the medical risks or the moral questions.
The side for genetic engineering won (as measured by convincing people).
LikeLike