A/B testing in politics
As research for my book I’m studying the way people use big data techniques, mostly from the marketing world, in politics. So naturally I was intrigued by Kyle Rush’s blogpost about A/B testing on the Obama campaign. Kyle was the Deputy Director of Frontend Web Development at Obama for America.
In case you don’t know the lingo, A/B testing is a test done by marketers to decide which of two ad designs is more effective – the ad with the dark blue background or the ad with the dark red background, for example. But in this case it was more like, the ad with Obama’s family or the ad with Obama’s family and the American flag in the background.
The idea is, as a marketer, you offer your target audience both ads – actually, any individual in the target audience either sees ad A or ad B, randomly – and then, after enough people have seen the ads, you see which population responds more, and you go with that version. Then you move on to the next test, where you keep the characteristic that just won and you test some other aspect of the ad, like the font.
As a mathematical testing framework, A/B testing is interesting and has structural complications – how do you know you’re getting a global maximum instead of a local maximum? In other words, if you’d first tested the font, and then the background color, would you have ended up with a “better ad”? What if there are 50 things you’d like to test, how do you decide which order to test them in?
But that’s not what interests me about Kyle’s Obama A/B testing blogpost. Rather, I’m fascinated by the definition of success that was chosen.
After all, an A/B test is all about which ad “works better,” so there has to be some way to measure success, and it has to be measured in real time if you want to go through many iterations of your ad.
In the case of the Obama campaign, there were two definitions of success, or maybe three: how often people signed up to be on Obama’s newsletter, how often they gave money, and how much money they gave. I infer this from Kyle’s braggy second sentence, “Overall we executed about 500 a/b tests on our web pages in a 20 month period which increased donation conversions by 49% and sign up conversions by 161%.” Those were the measures Kyle and his team was optimizing on.
Most of the blog post focused on getting people to donate more, and specifically on getting them to fill out the credit card donation page form. Here’s what they A/B tested:
Our plan was to separate the field groups into four smaller steps so that users did not feel overwhelmed by the length of the form. Essentially the idea was to get users to the top of the mountain by showing them a small incline rather than a steep slope.
What I find super interesting about this stuff (and of course this not the only “data science” that was used in Obama’s campaign, there was a separate team focused on getting Facebook users to share their friends’ lists and such) is that nowhere is there even a slight nod to the question of whether this stuff will improve or even maintain democracy. They don’t even discuss how maintainable this is.
I mean, we gave the Obama analytics team lots of credit for stuff, but in the end what they did was optimize a bunch of people’s donation money. Is that something we should cheer? It seems more like an arms race with the Republican party, in which the Democrats pulled ahead temporarily. And all it means is that the fight for donations will be even more manipulative, by both sides, by the next presidential election cycle.
As Felix Salmon pointed out to me over beer and sausages last week, the problem with big data in politics is that the easiest thing you can measure in politics is money, which means everything is optimized to that metric of success, leaving all other considerations ignored and probably stifled. And yes, “sign ups” are also measurable, but they more or less correspond to people who will receive weekly or daily requests for money from the candidate.
Readers, please tell me I’m wrong. Or suggest a way we can measure something and optimize to something that is less cynical than the size of a war chest.
mathbabe 
The easiest thing to measure in politics is votes. If you try to find a correlation between votes and money spent, well you will quickly learn your hand wringing is unnecessary.
LikeLike
Incorrect. Votes only get counted once. You can’t A/B test on them.
LikeLike
Actually you can, and do, A/B test on proxies for votes. Look at what groups like the AFL-CIO and Analyst Institute are doing. Since at least 2006 they’ve been doing in cycle tests of mail, phone, etc contact for voter persuasion and turnout.
It’s an entirely different part of the organization doing it than the fundraising part, and happens much more offline because that’s where voters it’s worth talking to are.
LikeLike
Thanks for the lead I will look into this.
LikeLike
If the dark red background is correlated with getting more money, and more money is correlated with getting more votes, it doesn’t follow that the dark red background is correlated with getting more votes. (Though admittedly you can be excused for guessing so if you don’t have any more information than that.)
LikeLike
“the fight for donations will be even more manipulative”
Welcome to democracy!
LikeLike
I know I’m being far too optimistic here, but if you have an engaged, money donating audience already at your site, why not measure policy preferences in a survey? It might be a moot point for a lame duck like Obama (or maybe not) but for someone just creating a platform it seems ideal. Surely nasty things like weighting preferences by donation amount would happen, but isn’t that already our flavor of democracy?
LikeLike
I guess leading and educating is right out. 😦 So just get as much money from people who care deeply about issues that may not matter in the real scheme of things. Blech. I am generally incredibly optimistic on most matters. But regarding this cesspool, I am afraid it is decline decline decline and then, maybe one day, some sort of revolution.
Want to get really depressed? Watch Ted Cruz’s campaign announcement at “Liberty” (how they get away with that title….) yesterday. He asks very early on for us to imagine (a big word throughout — reminds me of “hope”) imagine a country where not just 1/2 of all born again Christians vote but where all born again Christians vote and their values become the way of the land. I did imagine that and spent the rest of the day huddled in a dark room. I saw that 55% of Americans would favor a constitutional amendment making the US a Christian nation.
LikeLike
I hope it encouraged you to never miss a vote ~ and do some of your own A/B testing on F&F to get them to the polls as well.
LikeLike
A constitutional amendment establishing Christianity? The archbishop of San Francisco? I can’t hardly wait! 😉
As for Christian values becoming the way of the land, that would be a refreshing change. Bring back usury laws, for instance. 🙂 Feed the hungry. Shelter the homeless. 🙂
LikeLike
Ha. If the values were anti-usury laws, feeding the hungry, and sheltering the homeless, that would be fine. I think, however, “Christian Values” is code for far more than that. 🙂
LikeLike
Well, if all born again Christians voted, Cruz might be surprised. Jimmy Carter is a born again Christian, for instance. 😉
LikeLike
On the assumption that the plutocratic class does not donate via web sites, I think that maximizing web site donations weakens the power of that class and thus strengthens democracy, if only to a small extent.
LikeLike
Cathy, I am not a mathematician. But I will take a shot at this from an unsophisticated point of view. This is not about democracy, or anything except collecting donations. Don’t think these idiots care what the numbers say as long as they get the money. How far off am I?
LikeLike
Your post reminds me of a question my 7th grade teacher asked. We had been studying newspapers, and finally one day she asked, what is the purpose of a newspaper? Most students, including myself, responded with something like, “to inform people of the news”. She went around the room calling on students, and it became apparent these answers while they weren’t necessarily wrong, weren’t really right. Finally one student said, “to make money”. That was what she was looking for. And the answer completely surprised me, but also immediately made sense.
The Obama web team (or whatever its called) has one job only, and that is to design things to bring in money (well, maybe I am oversimplifying).
LikeLike
People always make money out to be evil, but it is a great normalizer. It works well for trade, but also for normalizing loss or gain across many things. It’s really just converting to a common unit of measure. Of course, the conversion factors can make you look good or evil even when mathematically accurate.
LikeLike
> What I find super interesting about this stuff … is that nowhere is there even a slight nod to the question of whether this stuff will improve or even maintain democracy.
Precisely. This is a big problem for the Democratic party – deciding whether to function as hip socially-liberal marketeers or as a more fuddy-duddy populist political organization focused on unsexy things like labor, foreign policy, and environmental issues.
LikeLike
Why stop at tailoring the ads for money: adjust policies for increasing donations.
Of course that’s how it really works – it’s where the big donors are. The ads are for hoovering dumb money.
LikeLike
There was a great series from the Daily Show when John Oliver went to Australia to discuss how they passed gun control legislation there. And in one of those bits, he asked Australian lawmakers what the point of being a legislator was, and they said “To serve the public”. He asked an aide (to Harry Reid, I believe) and the answer was “To get re-elected”. When John asked him if he was sure that was the reason, he thought a bit and basically hedged himself with some talk about serving the public, but it was an incredibly telling moment. Our political system is not designed to produce the best outcome for the public, it’s designed to produce the best outcome for the ones who are joining it and their donors.
LikeLike
holy. crap.
LikeLike
So, not that I want to entirely defend spinelessness. But I think an element of political calculation is reasonable, and something that’s worth debating over inside the party.
Say the Senate Dems had put up a really strong, Australia-like bill, after Newtown. And the whole caucus, including members in vulnerable districts, voted for it, and even either found a couple Republicans willing to vote for it, or found some procedural method (a really strained use of reconciliation, or the “nuclear option”) to break a filibuster.
And then the Republicans in the House refused to pass it anyways, and every single vulnerable Dem, and those moderate Republicans, all lost in the next election.
Was this action useful? Did it save any lives? Did it make Americans better off on net?
I think you can make a reasonable argument that having people stand on this principle repeatedly MIGHT help, in the long run, in terms of moving the Overton Window. But it’s not actually clear that it has that effect; in fact, standing for more moderate legislation, versus standing for strong legislation that FAILS UTTERLY, and destroys the political careers of a bunch of the people who voted for it, might spook a bunch of potential advocates and set the cause back.
While that aide to Reid put things inartfully, he does have a point: You can’t govern if you don’t win. So you have to think long and hard before you give up a win for the stake of making a rhetorical stand, which is quite different from risking your seat for an actual policy change. Quite a few Dems did risk their seats (and some probably lost their seats) for Obamacare. And the leadership pressured them into standing with the party on that. So it’s not like the Dems can’t stand for anything ever.
I tend to believe that they don’t take that kind of risk as often as they should. For instance, I think the Dems should’ve opened their negotiations over the ’09 stimulus package with Shock and Awe — they should’ve gone for $1.2T package, heavy on Dem priorities (infrastructure, aid to state gov’ts to keep schools and such healthy, a _huge_ FICA rebate, a big topping-up of unemployment insurance funds, etc). I think that negotiating from there, losing some size and trading a few pieces for Republican-friendly business tax cuts, would’ve ended up with a stronger package. Instead, they opened with something that was already small, and that already included about 30-40% items from the GOP agenda — as Bush once put it, they were negotiating with themselves.
But, well, maybe I’m wrong — maybe the public would’ve viewed a strong opener as craziness, and the whole thing would’ve crashed and burned, embarrassing everyone, and then they would’ve either gotten nothing (total disaster) or they would’ve had to come groveling back with an even more GOP-friendly package.
I may like the Elizabeth Warrens of the world more than I like Reid or Schumer — but I think the party needs some Reids and Schumers thinking about these things.
LikeLike
I think your reply misses the biggest problem with the American political system: We only have two parties, both of whom prefer big money donors to good policy outcomes. For each Elizabeth Warren, there are two (or more) Joe Manchins. Even under your own analogy, for each Warren, there is a Reid and Schumer — and that’s just within the Dems! Now take a Republican party which has gone off the deep end, and a system which automatically rewards being a member of either of the two major parties, and we end up with politicians whose policy stances are: “I’m not a member of the party”. Very rarely do we see politicians define their beliefs, because they know they need to get into their cash cow — er, office — first before they can begin to go against their campaign promises.
The point being: political calculation is hardly unique to US politics. But when something truly tragic happens — like, say the murder of school children — we lack the political machinery to do something about it, because every politicians’ first instinct is “how does this play to the base” and not “how can we make society better for everyone”.
LikeLike
You’re not wrong. This is why I’m a big fan of reforms that help shake up the duopoly. In CA, our Secretary of State Debra Bowen, who was a grassroots favorite when she first ran, helped push through a law that makes it easy for cities and counties to experiment with different electoral systems (particularly transferable vote type things, and proportional representation for local legislative races). I’m not convinced that PR is a cure-all — look at how it plays out in Israel — but I do think it could help in some ways.
I think if you care about these issues, for now, you’re a lot better off trying to win reforms from within a major party than you are saying “a pox on both your houses”. There’s the possible exception of a few metro areas where third-party or independent candidates can actually win — but even there, if you believe in localities as the “farm team” for higher offices, as Laocoon discusses below, really you’re probably even better off if you can push iconoclastic, principled candidates across the line in a major party primary.
Maybe we’d be better off if the calculating types were only serving as advisors, and the actual elected figures were more idealistic, and could make decisions about when to listen to words of caution, and when to stake their careers on principle…
LikeLike
Apropos of this conversation:
http://www.nationaljournal.com/magazine/rnc-chief-katie-walsh-20150327
“If you can get people to give money,” [the RNC finance director] tells me when I visit her at RNC headquarters, “that’s a much bigger get than getting people to vote for you.”
Nice to know that we’re formally stating that money matters more than votes, now. 😛
LikeLike
Holy crap. Thanks.
LikeLike
I’ve observed additional knock-on effects of the structuring of political campaigning based on big data in politics. The following bode poorly for political participation in future election cycles.
1) In the 2012 election, the pivotal counties that would tip for Obama were identified. Get Out The Vote efforts were targeted narrowly at Democratic voters in these counties. It’s generally accepted that it takes 7 contacts to get a voter out. So this requires a huge concentration of volunteers to make the calls. In the red state where I live (where nothing could have been done to get Obama or any Democrat elected), our local volunteers were directed to campaigning/calling in nearby North Carolina and to calling voters in Florida.
2) In the past several election cycles, the A/B and similar email response preference studies were done and we party loyalists were bombarded with the donation solicitations you describe. The big candidates or organizations like MoveOn collected a lot of money. The more in your face they were with sensationalized emails, the more people donated. (I took my own name off these lists, but my conversations with other local activists enlightened me that they were giving like crazy.)
While these may seem like great strategies, they are narrow tactics that work short-term for large scale campaigns. STATE AND LOCAL CANDIDATES WERE COMPLETELY IGNORED. In fact, 1 & 2 above completely cannibalized the down-ballot races in favor of short-term gains for the high-profile race. Volunteers and cash were stripped from the local races that identify up-and-comers, re-position the party in voters’ minds, and build the pipeline for continued success.
1 & 2 are driven by the data, so this is the tail wagging the dog. Stemming from this, the success of local candidates (low data) has been thrown to grass-roots organizers, who are typically under-resourced and distracted by multiple competing causes. Sort of like, we’ll bankrupt you, ignore you, and then blame you for not being successful.
This is a very real consequence of over-focus on data (because it’s there). The over-focus narrowed the range of options and completely eliminated additional considerations of building a bench of future superstar candidates. I totally understand that campaigns are very short-term in outlook, planning and execution. However, they have eaten their own lunch, and very few are seeing the consequences yet.
LikeLike
Great points, thanks for commenting!
LikeLike