On trusting experts, climate change research, and scientific translators
Stephanie Tai has written a thoughtful response on Jordan Ellenberg’s blog to my discussion with Jordan regarding trusting experts (see my Nate Silver post and the follow-up post for more context).
Trusting experts
Stephanie asks three important questions about trusting experts, which I paraphrase here:
- What does it take to look into a model yourself? How deeply must you probe?
- How do you avoid being manipulated when you do so?
- Why should we bother since stuff is so hard and we each have a limited amount of time?
I must confess I find the first two questions really interesting and I want to think about them, but I have a very little patience with the last question.
Here’s why:
- I’ve seen too many people (individual modelers) intentionally deflect investigations into models by setting them up as so hard that it’s not worth it (or at least it seems not worth it). They use buzz words and make it seem like there’s a magical layer of their model which makes it too difficult for mere mortals. But my experience (as an arrogant, provocative, and relentless questioner) is that I can always understand a given model if I’m talking to someone who really understands it and actually wants to communicate it.
- It smacks of an excuse rather than a reason. If it’s our responsibility to understand something, then by golly we should do it, even if it’s hard.
- Too many things are left up to people whose intentions are not reasonable using this “too hard” argument, and it gives those people reason to make entire systems seem too difficult to penetrate. For a great example, see the financial system, which is consistently too complicated for regulators to properly regulate.
I’m sure I seem unbelievably cynical here, but that’s where I got by working in finance, where I saw first-hand how manipulative and manipulated mathematical modeling can become. And there’s no reason at all such machinations wouldn’t translate to the world of big data or climate modeling.
Climate research
Speaking of climate modeling: first, it annoys me that people are using my “distrust the experts” line to be cast doubt on climate modelers.
People: I’m not asking you to simply be skeptical, I’m saying you should look into the models yourself! It’s the difference between sitting on a couch and pointing at a football game on TV and complaining about a missed play and getting on the football field yourself and trying to figure out how to throw the ball. The first is entertainment but not valuable to anyone but yourself. You are only adding to the discussion if you invest actual thoughtful work into the matter.
To that end, I invited an expert climate researcher to my house and asked him to explain the climate models to me and my husband, and although I’m not particularly skeptical of climate change research (more on that below when I compare incentives of the two sides), I asked obnoxious, relentless questions about the model until I was satisfied. And now I am satisfied. I am considering writing it up as a post.
As an aside, if climate researchers are annoyed by the skepticism, I can understand that, since football fans are an obnoxious group, but they should not get annoyed by people who want to actually do the work to understand the underlying models.
Another thing about climate research. People keep talking about incentives, and yes I agree wholeheartedly that we should follow the incentives to understand where manipulation might be taking place. But when I followed the incentives with respect to climate modeling, they bring me straight to climate change deniers, not to researchers.
Do we really think these scientists working with their research grants have more at stake than multi-billion dollar international companies who are trying to ignore the effect of their polluting factories on the environment? People, please. The bulk of the incentives are definitely with the business owners. Which is not to say there are no incentives on the other side, since everyone always wants to feel like their research is meaningful, but let’s get real.
Scientific translators
I like this idea Stephanie comes up with:
Some sociologists of science suggest that translational “experts”–that is, “experts” who aren’t necessarily producing new information and research, but instead are “expert” enough to communicate stuff to those not trained in the area–can help bridge this divide without requiring everyone to become “experts” themselves. But that can also raise the question of whether these translational experts have hidden agendas in some way. Moreover, one can also raise questions of whether a partial understanding of the model might in some instances be more misleading than not looking into the model at all–examples of that could be the various challenges to evolution based on fairly minor examples that when fully contextualized seem minor but may pop out to someone who is doing a less systematic inquiry.
First, I attempt to make my blog something like a platform for this, and I also do my best to make my agenda not at all hidden so people don’t have to worry about that.
This raises a few issues for me:
- Right now we depend mostly on press to do our translations, but they aren’t typically trained as scientists. Does that make them more prone to being manipulated? I think it does.
- How do we encourage more translational expertise to emerge from actual experts? Currently, in academia, the translation to the general public of one’s research is not at all encouraged or rewarded, and outside academia even less so.
- Like Stephanie, I worry about hidden agendas and partial understandings, but I honestly think they are secondary to getting a robust system of translation started to begin with, which would hopefully in turn engage the general public with the scientific method and current scientific knowledge. In other words, the good outweighs the bad here.
mathbabe 
The idea of translation experts is an interesting one. A portion of my role as a central bank economist is just that…I help critically evaluate external and internal studies in my subject area. This analysis takes many forms, but one thing that has always struck me is that the translation work does not reach a broader audience. There is a clear internal demand and that is met first, but it seems like the output could serve a secondary audience too. Yet, it’s not even clear to me where the external demand is (or if it exists) and how to best transmit that information. And no, I can’t just start my own blog…some jobs are “a blessing and a curse.”
LikeLike
> How do we encourage more translational expertise to emerge from actual experts? Currently, in academia, the translation to the general public of one’s research is not at all encouraged or rewarded, and outside academia even less so.
The default reward in currency is money.
Actual experts can be amply rewarded for translational work by doing public-facing teaching and writing public-facing books if they’d learn a bare smidgen about doing such activities economically, so as to be indefinitely sustainable.
That’s why I teach online courses DIY. I.e. the way self-published authors sell their own books, even though the label of “expert” doesn’t quite fit me. Maybe I can serve as an example for the “actual experts”? 🙂
> But that can also raise the question of whether these translational experts have hidden agendas in some way.
This and the also-mentioned problem of misleading translation won’t matter as much if only their numbers would grow. Let’s put out faith in a literate population equipped with workable BS-detectors. They’d readily sift, winnow, and thresh if only there were enough of a harvest to begin with!
LikeLike
Interesting piece – thought I would add my $0.02 on the scientific translator idea. My experience (pharmaceutical development) bridges the often ambiguous world of science with the black/white world of business. In the business world, a problem either is, or is not solved whereas the scientist thinks in terms of probabilities.
I suspect many of the layman climate-deniers also tend to be black/white thinkers (leaving aside those deniers who have financially driven agendas) who will change their minds once anthropomorphic climate change is ‘proven.’ The science is there but translating climate model probabilities into absolutes is a challenge, so perhaps specialized scientific translators could be the bridge.
LikeLike
In academia, a “translational expert” can be a reviewer, member of a grants committee, or head of a department. Thank goodness the system works better than it does in finance, but it is not perfect. Take, e.g., the “Nobel” prizes in Economics.
LikeLike
True, however these same experts are not adequately translating to the public (see below). Also, the public often gets conflicting messages – e.g. coffee is good for you; coffee is bad for you; coffee is good for you again – that’s the nature of research / discovery but the impact is loss of credibility.
LikeLike
I don’t think your 3. is the same as Steph’s 3. — hers doesn’t say “model checking is hard and time-consuming so why bother?” It is more like “model checking is hard and time-consuming, thus each one of us, no matter how devoted to model-checking, is going to leave a lot of models unchecked, so what should we do about the questions to which the unchecked models apply?” More concisely: the time you spend playing football is time you are not spending playing tennis.
LikeLike
Hey, I only found this very belatedly, but did want to comment even though it’s so late. I really don’t want anyone to get the impression that I think “model checking is hard and time-consuming, so why bother?” Indeed, I spend a lot of time working with scientists in communicating models to the public, and a lot of time researching and working on better public participation mechanisms so that nonexperts (as we all are in some area or another) can be involved in public decisionmaking: two things I wouldn’t do if I thought “why bother.” So it’s upsetting to me to see what I wrote characterized in this manner.
Instead, Jordan’s characterization is far more accurate, that given the finite time we have in this world, and the amount of information that there is in the world, there’s just going to be stuff that each individual has no time to get to. This absolutely *does not* mean I think we shouldn’t try.
LikeLike
Steph,
Sorry, you’re right. It was an unfair characterization of what you said.
Cathy
LikeLike
Thanks, I really do appreciate that. For me, it’s actually less the fairness and more that a lot of the volunteer work that I do do is about risk communication (both in terms of scientists communicating to the not-necessarily-scientific legal audience and in terms of the not-necessarily-scientific legal audience taking the time to understand scientific models), so this hit particularly at something quite personal.
I agree on a personal level that we should all individually try to do what we can to better understand the world. But what I work on as now as a legal scholar has to do with creating/reforming systems surrounding an imperfect, but existing, world–one where everyone doesn’t take that time (and indeed, in many ways, as individuals cannot take that time for every single issue out there, although I’d like there to be more group-sourced ways of decisionmaking such that that information is “in there” somehow.)
LikeLike
“But my experience (as an arrogant, provocative, and relentless questioner) is that I can always understand a given model if I’m talking to someone who really understands it and actually wants to communicate it.”
The danger one has to guard against, though, is concluding that a person either doesn’t understand the model themselves or doesn’t want to communicate because it’s hard to make out what they’re saying. I keep coming back to math — what if somebody from outside the discipline wanted me to explain the math I was working on, thoroughly enough so that they could form an opinion as to whether my proofs were correct? No matter how hard and how sincerely I tried, there would be no way I could accomplish this in any reasonable amount of time.
It may be that financial modeling and climate models are much, much simpler than number theory, but I don’t know why I should default to thinking that’s the case. And I do know that related questions about validity of inference, like the ones the experts tussle over on Gelman’s blog, are really difficult and subtle. If I sat down with Gelman for a whole day of probing questions would I come out of it with a strong and justified sense of confidence that his approach was correct? Surely not — and that’s not because he wouldn’t authentically be trying to communicate.
On the other hand, sometimes it’s easy to figure out that something _isn’t_ correct. This aids my confidence in climate science, because one so frequently reads arguments by climate optimists that are plainly wrong. You say to yourself: if this is what they’re leading with, there can’t be much to it. It’s kind of like when the trailer for a comedy has no funny jokes, you can pretty confidently conclude that the movie itself isn’t funny. I think this works well for movies but I’m less confident in it as a methodology for belief formation on scientific topics.
LikeLike
Some of the best ‘translation experts’ are also ‘gatekeepers’. One that comes to mind is the late great C.P. Snow.
LikeLike
As a sort of member of the “press”, over the years I’ve worked with quite a few scientists in a variety of fields. Some are good at explaining their models, some not so good, and some simply don’t give a damn, essentially taking the position that if you can’t understand it, tough shit.
At the risk––in fact the certainty––of being overly simplistic, I’ll offer some thoughts regarding my own perspective on communicating complex models to the general public.
First, isn’t a model, at it’s root, simply a description of how the world works? That is, some slice, some aspect of the world? Admittedly the more subtle and abstract the aspect being described––complex financial market models being a good example––and the more divorced it is from everyday human experience, the less it may seem like a description of how the “world” works, that is the world in which we get up in the morning, have coffee and breakfast, get ourselves to work, and do our jobs.
But still, aren’t all scientific modelers positing that the model they are using is an accurate description about how that particular aspect of the world works?
So, back to the real world. I realize I’m far from the first to say this, but we are all modelers. We all build our own models of how the world works, otherwise we couldn’t get out of bed and get through the day. These models, I think, may be more or less “scientific”, in one important sense––either they are based on evidence, or they are based on someone else’s world view, a person to whom we have ascribed “expert” status.
Now, before you jump on me and say we all rely on experts, of course we do. The point is that the further away any issue gets from our own “expert” experience, the more we need to rely on other experts. I am an expert, for instance––from my own direct experience––on making my morning coffee the way I like it. If I grind the beans at the right setting, put the right amount into the filter, I’ll get coffee that tastes the way I like it. I am not an expert at roasting the coffee beans, however, and count on those who are to do that part of the process.
In this case, it’s pretty easy for me to acquire “evidence” that this other expert––the coffee roaster––knows what she’s doing. By a simple heuristic process, I find a brand I like, and stick with it as long as it’s consistent.
In the case of climate change, drug trials, or financial transaction models, it’s of course a lot more difficult for the non-expert to acquire the evidence, and even if it’s acquired, to make heads or tails of it. And I am quite aware that nothing I’m saying here actually makes that task any easier.
So to get back to my long winded point. From my translational journalistic perspective, Give Me The Evidence! Like the annoying Cuba Gooding character was always saying to the annoying Tom Cruise character in “Jerry Maguire”––Show Me The Money!
When, as a journalist/filmmaker, I am interviewing a scientist, my experience is that if he or she is good at providing evidence at two levels: a) what is the evidence of how the world works that led them to formulate their model, and b) what is the evidence that their models works as an accurate predictor of future events, then I am usually able to take that information and translate it into an interesting and understandable format.
As a journalist/filmmaker, evidence is good. Evidence is illustration. Evidence is grinding the coffee grounds correctly and putting the right amount into the filter, but knowing how I can go up one expert level to trust the coffee beans themselves.
In my experience, those scientists who fall into the “tough shit” category I described in the opening paragraph usually are unwilling (or perhaps, for good reason, unable) to provide either level of evidence.
So, to bring this screed to a close, here’s my overly simplistic advice to scientists and translators of scientific models: Like the business advisor, who when asked the three most important things to success for a retail storefront said “location, location, location”, I think the three most important things in translating a scientific model for the general public are “evidence, evidence, evidence”.
LikeLike
We definitely need more translators and more translations! But in addition to the right incentives, we also need a new format for writing, that aims to bridge the gap between a journalistic/popular science account of the subject and the research level technical articles that experts on the subject use to communicate with each other.
Suppose I want to understand current models of climate change and I don’t have access to an expert that I can engage in dialogue with. What can I do to understand the model? Reading journalistic articles or popular science books won’t tell me much about the actual model, even if they are well-written, just because the explicit purpose of such writing is to abstract away from the details. Delving into current research articles on climate change won’t help, because I do not understand the language they use. So I am left with the option of reading a couple of (under)graduate textbooks on related subjects to prepare myself for reading the actual articles – which is just not going to happen. Even if I am serious about playing football, I will not spend weeks lifting weights before I pick up a football.
What is missing is a form of writing that provides an accessible introduction to the non-expert, that does not try to abstract from the details, but instead encourages readers to get their hands dirty. It should allow readers to get a concise overview of the subject matter and then provide a path for them to do the actual work of going into depth. While no static document can ever substitute a dialogue with an expert (who can meet you at your individual level), I still think there is a huge space for introductory technical writing that needs to be filled – and I, for one, would love to read that sort of thing.
In some instances, blogs come closest to offering that kind of writing. One example that comes to mind are the posts on Paul Krugman’s blog that he classifies as “Wonkish”. Tellingly, they give a much better impression of the underlying models than, say, his most recent book. Still I think there is a lot of room to improve upon this kind of writing, especially with a view towards readers who are willing to put in work and want to understand the details.
LikeLike
I agree, and would like to add on a derivation to your comment—by substituting “reading” for “writing.” As in, “there is a huge space for technical reading that needs to be filled.”
I recently worked with some high school English classrooms to introduce them to what a peer-reviewed scientific paper looks like (a recent 5-pager in health sciences). We walked through the structure then compared the paper’s conclusions with a blurb written about it in a men’s health magazine (drink green tea for a better memory).
It was amazing how well the Jrs and Srs picked up the paper’s conclusions and located weak points in the translation to the magazine. I realized that they would not be able to critique the “model,” or experimental methods in this case, but—to follow the analogy—maybe they’re learning the rules of football while getting a tour of the stadium and locker rooms, and leaving with the keys to come back whenever they feel like (?).
I wanted to leave the students with a few thoughts:
(1) primary literature is written by scientists for scientists in that field, so use resources (textbooks, wikipedia, author’s email) where needed.
(2) you CAN usually understand the main points of most papers.
(3) Google Scholar is a great resource.
So, please forgive me, as I can see that this isn’t totally related to the original blog post, though at least some of these high schoolers will now understand the importance of the translation of science and have access (through GScholar, confidence) to the *evidence* that another commenter emphasized.
LikeLike
I think the process of making information digestible for public consumption is similar to making a movie. You need the story, than you need the money, then the screenplay then the director and most important the actor/actress to actualize,empower and make digestible the story for public consumption.I think the ‘right’ actor/actress is everything!
LikeLike
I highly recommend (1) Steven Epstein’s work* on how ACT UP activists became “experts” on all sorts of bio-medical matters that are highly technical. Most had little or no training, they became expert enough to sit on NIH panels. What they did have were friends and lovers who were dying. I think this is a precursor to the Occupy Alt Banking group in many ways.
(2) Ed Tufte’s work on data graphics and how they can be used as tools to think with among lay audiences. His entire enterprise is premised on communicating complexity not dumbing it down and on the assumption that regular citizens are quite bright enough to understand more than we suspect.
And, (3) lest you should think you are inventing the wheel here, this problem of how expertise and authority enter into democratic politics is a central theme in American political thought going back to disagreements in the 1920s between John Dewey and Walter Lippmann about democracy and the competence of citizens. This debate is really germane to your concerns.
______
* He has a really smart book IMPURE SCIENCE from a decade or so ago.
LikeLike
Just a quick thought: Who exactly are “lay people” in this context? The target audience for “translated expertise” certainly includes people with significant academic training, including active researchers in their own fields. Does this fit the generic image of a “lay person”? What type of translation is best suited to make our own field accessible to experts in other fields?
LikeLike
A key question. For me, lay people means John Q Public – public opinion because public opinion can shape policy. In practice, this only matters for a select few issues, such as climate change or science education. There is no impact to a lay person not understanding / believing in the Higgs particle for example.
LikeLike
I hate to play the part of Ancient Greek/Philosophy expert, but Kathy, you have a mis-guided view of what skepticism is. It is not casually dismissing things. It would include your engaging of modeling as part of a wider philosophic way of dealing with the world. I realize there is a modern corruption that thinks otherwise and imagines skepticism is simply dismissal or perhaps doubt, but really, skepticism is a philosophy that encourages ALWAYS doing the heavy lifting and taking little on faith.
I agree that the idea of scientific translators has interest. In fact I read your blog because you engage in discussion that has elements of this idea, as you rightly state. I think the last comment on your previous post in this thread, that suggested that how such a person comported themselves contributed enormously to their ability to reach people, was very helpful in thinking about this. Scientific translators will be believed on the basis of their humanity is what the quote from Onora O’Neill seems to me to say. And that has important lessons for scientists and their actions as “experts”.
Your three bullet points at the end, especially the first two, raise a number of thoughts for me. The problems with the press, even the so-called “elite media”, highlight the need for better math+science training at elite universities. You might have Larry Summers’s picture up for his complicity in financial shenanigans, but he was (IMO, anyway) right on for trying to push Harvard College to increase its requirements for math+science. Elite journalism is massively an Ivy League proposition these days and as long as those Ivy Leaguers get that bully pulpit, they should have to earn it and not hide from these subjects so much. At the very least, requiring them to take more math and science with create more variance in their grades and remove the conceit at say, Harvard, that 90+% of them deserve honors.
For developing translators, I think we need to realize that an approximate understanding is likely superior to none at all. Given the large number of underemployed scientists, I don’t think it is all that hard to find someone who might have unique, but not quite ‘world-class’ knowledge of say, climate modelling. What is hard, and I what I wished there were more of, were people who had some knowledge of the discipline, but who also understood the importance of those things Onora O’Neil was discussing, the very important human aspects that help a message to get through to people. Though even harder than finding these ‘sensitive scientists’, is finding someone who would pay them to do the translations!
LikeLike
Regarding number 3: you belittle it, but you make a serious mistake by doing so. It is true that most people are capable of understanding any *single* model (or issue). But there is an inexhaustible supply of important issues, and your time is finite.
LikeLike
There are some things that I observe myself (phenomenae). So, for those I need not ask: “What happened?” of someone else. For events where I was not present or doing telephone/imagery “surveillance”, I have to rely on the accounts of others. Some starting questions in “one-person investigations” might be: What did Herodutus do and how do I know it? or: What did Archimedes do and how do I know it? I think it’s worthwhile in many cases to examine the background (biographical details) of an expert or panelist: one can get an idea of their strengths and weaknesses, and also character. Also, in expert topics, there is generally an intellectual heritage in the “history of ideas”, so reading references in an article as well as around those references can help to acquire a bit more of that “intellectual heritage”. It seems to me that Stephanie Tai’s point is close to: “Given my finite resources, for any given question, at some point I might have to rely on expert “witnesses”.” This makes sense in judicial and administrative proceedings, which cannot go on indefinitely before issuing a report or decision. Cathy and others are more for “doing the heavy lifting” by oneself. I view the discovery of truth as an on-going process. At any point, in the furtherance of making the truth emerge, myself and others can write articles, in good faith, which tackle what I make of what’s known on some narrow question. For me, it’s important to explain how I arrive at my conclusions; this often involves citing various sources, what they said and where, and giving my general assessment about the author. If I’m candid about what I think I know, then it might even improve the dissemination of truth, whatever that is. For me, it’s not necessary to establish definitely the expertise of some person in some narrow area. What matters more is that my writing reflect accurately what actually went on in my mind when I was considering what’s known about a given question. Then others can pick up my reflections where I left them off, or go off on a tangent: it’s a collaborative, loosely networked group of people; modern computer communications can in principle help.
There’s the question of the target audience for the writer. I think that’s a difficult question. But one can write at different levels of exposition for different target audiences, while keeping in mind that the more arduous the reading, the more arcane the specialized knowledge, the longer the article: the smaller the readership is likely to be.
LikeLike
Sonic Charmer’s take on complicated models is worth reading: http://rwcg.wordpress.com/2010/07/04/all-large-calculations-are-wrong/
It’s mostly about finance, but there’s a bit on climate models:
“It may be answered: how do I know the climate calculations are wrong? Now I can say this because probably unlike you, I’ve actually worked on climate models and know what they are and what goes into them. Or I can do as I’m doing here and point to the fact that giant teams of highly-paid quants can’t even get a single-number calculation – VaR – correct. One single number! So why would we expect, like, some grad student (‘overseen’, i.e. pointed towards some literature and a data set, by his professor) working in virtual ivory-tower isolation on sloppy C code for a summer to be able to generate any sort of accurate model of the fricking entire future climate of the entire fricking earth? Why would we look at the latter’s paper and go Oooh and Aaah and declare we should base policy on it? (And if you don’t think that’s what climate modeling is, then what exactly do you think it is?)”
See also this: http://rwcg.wordpress.com/2009/11/30/my-accidental-priesthood/
LikeLike
Interesting discussion. But I think that mathbabe is comparing apples and oranges when discussing climate change. Please, let’s leave the Know Nothings out from _both_ parties. That includes 99% of commenters on whichever blog, and 85% of bloggers, and about 100% of politicians. Only the scientists count, and only the ones who follow mathbabe’s advice: Work through the models.
Ground rules okay? Fine, so who are we left with out there? Well, 1) Michael Mann, and Tamino for sure. But also 2) Judith Curry and Hans Storch. And also 3) Ross McKitrick and Steve McIntyre. These are all people who are spending time and real effort trying to work through the numbers, and anyone who denies it is denying reality.
And as far as I know, not one of them is receiving zillions of dollars from huge corporations. Judith Curry pointed this out years ago: The really dangerous and effective skeptics are a small group of amateurs doing real scientific work in their spare time.
Now, which of these three groups is larger? I’m pretty sure that the first group is very considerably larger, certainly much larger than the third group. But that never much mattered to scientists. Let them work it out. If their models are good, nature will comply. Currently, as Nate Silver points out, nature seems to be trying to suggest a much lower climate sensitivity than the consensus of the models. Maybe more time is needed for that to become clarified. But anyhow, let the science work it out.
But mathbabe: “To that end, I invited an expert climate researcher to my house and asked him to explain the climate models to me and my husband … I asked obnoxious, relentless questions about the model until I was satisfied.” What would mathbabe have said if the researcher had refused to answer her questions, or refused to give access to the data when she wanted to try some tests of her own, or refused to give the exact algorithm code used when she wanted to try to replicate his work? Would she have been satisfied then, or would she have concluded that he’s an incompetent trying to hide really shoddy work? Anyone who follows Steve McIntyre’s blog gets an unending, intricately documented look at a decade-long struggle to try to figure out exactly how the paleoclimatology people got their results. He’s been doing _exactly what mathbabe says he should be doing_, and one can read post after painful post showing how the (1)-type scientists did their best to block him from trying.
LikeLike
The fact that you refer to climate change ‘deniers’ does not inspire confidence that you are approaching this issue with the rigour and objectivity that one would expect of a mathematician.
Most climate sceptics have no links with industry whatsoever, despite the relentless and dishonest smear campaign against them which you seem to have fallen for.
LikeLike
Possibly. But my experience with them does not warrant the term “skeptic”, which implies an open mind.
LikeLike