mathbabe

Papers have mistakes all the time. If they’re smallish mistakes that don’t threaten the main work, often times the author is told to write an erratum, which the academic journal publishes in a subsequent volume. Other times the problems are more substantial, and might deserve the paper to be retracted altogether.

For example, if a paper is found to have fraudulent data, retraction is called for. Even when the claims made are outlandish, implausible, and unreproducible, but the authors hadn’t been intentionally fraudulent, there still may be just cause to seriously question their claims and retract. On the other hand, if a paper that was once deemed cutting edge and new is, in retrospect, not very innovative at all, then typically no retraction is called for; the paper is simply ignored. When exactly retraction happens, and how, probably depends on the journal, and even the editor.

Today I want to tell you a story in which that process seems to have gone badly wrong.

Elsevier, the academic publishing giant owns a journal called the Journal of Computational Science (JoCS) which published a paper called Twitter Mood Predicts the Stock Market (preprint version here) back in 2010. It got a lot of press, and even more, and according to Google Scholar has been cited 1300 times. According to media reports, the paper showed that Twitter, when it was enhanced with emotional tags, was able to predict the Dow Jones Industrial Average with an accuracy of 87% (whatever that means).

Full disclosure: I haven’t read the paper, but even so I don’t believe the results of this paper. People in hedge funds have been trolling for signal in all sorts of news and social media text-based ways for a long while, and there’s simply no way that they would have ignored such a strong signal all the way into 2008. If it was real, they wouldn’t have ignored it, and it would have faded. But I also don’t think it’s so real either.

Anyway, that’s my personal intuition about this, but I could be wrong! That’s what’s cool about academic publishing, right? That we could just be super wrong and people can say what they think and then we get to have this open conversation?

Well, sometimes. What actually happened here is that a bunch of people tried to replicate these results, which was harder because suddenly Twitter started charging lots of money for their data, and a hedge fund also tried the Twitter strategy that was similar to the one outlined in the paper, but everyone lost money*.

After a while, one of these frustrated would-be traders, who we will call LW, decides to write a letter to the editor complaining about the original paper. He even blogged about his letter here. In his letter he had two complaints. First, that the results were consistent with datamining, which is to say that there’s statistical evidence the authors cherry picked their data. Second, that if the results were true, they would violate the “Efficient Market Hypothesis,” and would surprise a bunch of traders with many decades of experience.

So far, so good. A paper is published, people are complaining that the results are wrong or extremely implausible. This is what academic publishing is for.

Here’s what happens next. The editor sends out the letter to reviewers. Two out of 3 of the reviewers respond, and I’ve got a copy their responses. The first reviewer is enthusiastic about doing something – although whether that means retracting the Twitter paper or publishing the complaint letter in the “Letter To The Editor” section is not clear – and uses the phrase “The original paper’s performance claims are convincingly shown to be severely exaggerated.” That first reviewer has minor requests for modifications.

The second reviewer is less enthusiastic but still thinks there is merit to the complaint letter. The second reviewer is dubious as to whether the original article should be withdrawn, but is clearly also skeptical of the stated claims. Finally, the second reviewer suggests that the original authors should be given a chance to respond before their article is retracted.

At this point, the editor writes to the complaint letter writer LW and says, you need to modify your letter, at which time I’ll “reconsider my decision.” The editor doesn’t say whether that decision is to retract the paper or to publish the letter.

So far, still so good. But here’s where things get very weird. After modifying the letter, LW sends it back to the editor, who soon comes back with another review, and importantly, a decision not to take further action. Here are some important facts:

1. The new review is scathing, passionate, and very long. Look at it here.
2. The new review has a name on it – possibly left there by accident – it’s the author of the original paper!
3. Perhaps this was intentional? Did the editor want to give the original author a chance to defend his work?
4. In the editor’s letter, he states “Reviewers’ comments on your work have now been received.  You will see that they are advising against publication of your work.  Therefore I must reject it.”
5. The way that was phrased, it doesn’t sound like the editor was acknowledging that this was not an unbiased reviewer, but was in fact one of the original authors.
6. In any case, before the final reviewer weighed in, it looked like the reviewers had been suggesting publication of the letter at the very least, possibly with the chance for another reaction letter from the author. So this author’s review seems to have been the deciding vote.
7. You can read more about the details here, on the complaining letter writer’s blog.

What are the standards for this kind of thing? I’m not sure, but I’m pretty certain that asking the original author to be the deciding vote on whether a paper gets retracted isn’t – or should not be – standard practice.

To be clear, I think it makes sense to allow the author to respond to the complaints, but not at this point in the process. Instead, the decision of whether to publish the letter should have been made, with the help of outside reviewers, and if it was decided to publish the letter, the original author should have been given a chance to compose a rebuttal to be published side by side with the complaint.

Also to be clear, I’m not incredibly sympathetic with someone trying to make money off of a published algorithm and then getting pissed when they lose money instead. I’m willing to admit that more than one of these parties is biased. But I do think that the process over at Elsevier’s Journal of Computational Sciences needs auditing.

* Or at least the ones that are talking. Maybe other traders are raking it in but aren’t talking?