Crowdsourcing a Theranos test?
Have you been following the Theranos debacle? The WSJ reported twice last week on this much-hyped Silicon Valley company which is trying to “disrupt” the blood test industry but seems to be stumbling on fraudulent methods. The company has fancy investors and even fancier board members (former Secretaries of State George Schultz and Henry Kissinger and former Secretary of Defense William Perry) and was valued early last week at $9 billion. I’m pretty sure its value has gone way down since then. Since WSJ is behind the paywall, take a look at this summary on Forbes or this one from Wired.
Well, does the blood test technology work or not? It’s frustratingly hard to know. Theranos CEO and founder, Stanford dropout, and black-turtleneck-wearing Elizabeth Holmes claims (for example at the end of this Mad Money interview from last week) there have been multiple tests of their methods against standard (more expensive) tests that require more blood. But she doesn’t provide them to the public for scrutiny. So that’s unsettling.
Here’s my idea. We crowdsource the answer to this question. It’s not a random sample but that’s ok, because we already have “ground truth” in the form of standard tests. We just want to compare Theranos blood test results against them. This guy did it already:
On June 29th I went to the Hematology lab at Stanford for routine CBC and Metabolites numbers. As I walked back to Palo Alto, I stopped by my doctor’s office, got an order, went to the Theranos office at Walgreens on University Avenue in Palo Alto and got a CBC test.
Taken one hour apart, the Stanford and Theranos HCT numbers differ by about 7%: 44.1 Theranos vs. Stanford 41.1. For platelets, the difference is even wider: Theranos 430 vs. Stanford 320
Intrigued, I got a new order and went back to Theranos the following day, on June 30th. Theranos numbers were markedly different 24 hours later: HCT 40.6; PLT 375
Just to make sure, I went back to Stanford for a second test today July 1st: Stanford HCT 41.7; PLT 297
I find the price and convenience of Theranos services attractive, but I worry about the reliability of the important HCT number. What is the confidence interval in your measurement? + or – 1 point? + or – 5 points? I do get a phlebotomy at 45. How should I look at your June 29th 44.1 HCT number?
I’m curious to hear more about your methodology, standards and quality controls and would like to give you an opportunity to respond before I write a Monday Note on the broader topic of lab exams and other healthcare mysteries.
More people should do this, preferably on the same day! Within an hour of each other, too, if possible. It’s in the public interest. We just need to set up an app or something to let people upload their results with some kind of verification method so we know it’s not spammy.
Or else we just ignore Theranos entirely, because it’s gotta be a fraud given the way they’re acting. Here’s a convincing argument from the comment section of the above first person account, someone who calls themselves Skeptical Owl:
You are the CEO of a company that has been working on a revolutionary, disruptive technology for a decade or so. This technology is so amazing that, based on price and customer experience, you can capture most of the (very large) existing market as soon as you enter it. Armed with all of these advantages, you choose to avoid allowing scientists or regulators to validate your technology, enter the marketplace through a single partner (Walgreens) at a glacial pace, and conduct most of your business using existing technologies that are not your revolutionary product. Are you choosing this strategy because your technology doesn’t actually work, because you are incompetent, or because you hate capitalism? Bonus question: if the technology doesn’t work, why is your board a Who’s Who of the military-industrial complex instead of a group of scientists who can help?
Update: I just received this email from a Theranos PR firm:
We read your coverage of Theranos with interest, and wanted to share with you that – because there has been a lot of inaccurate information in the media to date – we have posted detailed information on our technology, finger-stick test, accuracy, and conversations with The Wall Street Journal on our website: https://www.theranos.com/news/posts/custom/theranos-facts
We hope you will take the time to review the information we have posted online, and look forward to engaging with you in the future.
Regards,
Peyton Burgess, on behalf of Theranos
FTI Consulting
mathbabe 
“…it’s gotta be a fraud given the way they’re acting”… that’s pretty strong language, but it’s my gut reaction as well (though it could involve sincere folks who duped themselves and got carried away). Despite following Wall Street and spending much of my working life in clinical genetics, somehow I completely missed the whole Theranos story until the WSJ story. And now trying to catch up on it, it’s hard to understand how the hype ever got so high (…hmmm, stock market bubble on the way).
LikeLike
“The company has fancy investors and even fancier board members (former Secretaries of State George Schultz and Henry Kissinger and former Secretary of Defense William Perry)…”
Vampires. Writers, get on this.
LikeLike
I suspect not fraud, but confirmation bias and culture mismatch. I’ve been chief engineer for regulated medical products, and I’ve dealt with startup inventors. The inventor has intense confirmation bias (finding good reasons to discount all problems) and comes from a culture of “when in doubt, try it”. For a regulated medical device they hit the walls of pessimistic FDA statistical reviewers who say “prove it, show me *all* the evidence both good and bad” with a culture of “when in doubt, do more studies and trials.”
The startups complain bitterly about unreasonable FDA processes. From my perspective the FDA bends over backward to educate, explain, advise, and reccomend approaches. The thing they don’t want to do is approve without evidence. Gathering the evidence and showing scientific and statistical validity remains a requirement on the startup and a totally new experience for the startup.
BTW the standard to be met is “as good as the worst already approved method”. You need not be better. The FDA expects the market to select for quality once it’s managed the minimum necessary entry bar.
LikeLike
Yes, I agree that it might not feel like fraud from the inside. That’s just what we outsiders see. After all, Holmes was quoted as bragging that when she was a kid (age 7) she had designed a time machine. No shit, this woman is a wishful thinking inventor.
LikeLike
We don’t really have ground truth for the standard tests, because (at least to patients) the variance in the measurements is not released.
LikeLike
So we should crowdsource that too. I mean, a simple thing to do would be to take the same test 5 times, or something, at one sitting, right? Presuming this doesn’t cause you to faint for loss of blood.
LikeLike
Perhaps someone with a clinical background could point to relevant literature for the existing tests, including: (a) fluctuations in key measurements throughout the day, (b) variation in key measurements across populations (attempting to control for daily fluctuations)?
There’s also a 10-15 year research trail of various groups working with microfluidics and blood (eg: the Ismagilov group, now at Caltech, had a functioning microfluidics model of clotting ~10 years ago).
LikeLike
It’s a lot of work to find this stuff, but look for the 510(k) or PMA filings for the predicate medical devices. These are public records. Details may be redacted from the public portion for proprietary disclosure reasons, but the basic criteria used should be public. You’ll need to do this for each individual test. They all have different criteria. Digging this stuff out is usually left to paid specialists because it’s a lot of searching, reading, and following references many times over. Experience and domain knowledge make it go a lot faster for them. But it’s all are public records, so anyone who wants to do the work has access.
LikeLike
Feel like a possible elephant-in-the-room here needs to be voiced out loud: you have a very young, intelligent, lissome blonde engaging middle-aged-and-above men who have money and influence, but probably little understanding of the technology she was touting… perhaps a recipe for disaster (not to say it couldn’t succeed, but just that great care ought be taken — as the saying goes, men don’t always think so clearly with their brains when they’re swayed by other body parts 8-/… Did she have some ‘name’ female backers, I’ve only seen male names?).
LikeLike
I definitely think her looks had something to do with it, even if it’s just the Steve Jobs impersonation. Also, shouldn’t someone on the board actually understand the technology of a company?
LikeLike
So on Sunday, my wife had a CBC and a CMP (in total 2 vials) which needed to be extracted and collected for the clinical trial that she’s on. It’s not easy extracting blood from her veins. First the tech tries, and draws a tiny amount and can’t do any more. Than a nurse tries, and more stabbing and another almost dry well, and no one knows how much blood is really needed for analysis. Then comes a third nurse, with 30 years experience, and she gets the job done. And all the while I’m thinking there has to be a better way, and when is Theranos going to perfect their method and make it available broadly.
I doubt that two serial tests done the classical way will give identical results, so I’m not sure what the proper benchmark should be for Theranos. But I am quite upset by the amount of sexist remarks in the comments section here. Would people write the same sexist scheissdreck if the founder were a man? Her looks? Really?
Do anomalies exist for the classical test? You be the judge.
In reverse chronological order (WBC):
03/27/2011 7.5 K/mcL
[4-11 K/mcL]
03/23/2011 31.9 K/mcL
[4-11 K/mcL]
03/22/2011 33.1 K/mcL
[4-11 K/mcL] RUN REPEATED
LikeLike
The WBC is not a good number to try to duplicate over several days, as it is a marker of the body’s response to infection and may vary widely from day to day depending on the health status of the patient. HCT, on the other hand, tends to stay relatively constant in the absence of bleeding or major changes in hydration status. PLT is somewhat more variable than HCT. So the best one to try to duplicate would be HCT. (I am a physician.)
LikeLike
Thank you, Doctor. I extracted the WBC data only because it said “run repeated,” and because I suspect that the blood sample may have gotten contaminated.
The HCT from the same period:
03/27/2011 33.5 %
[34-46 %]
03/23/2011 30.4 %
[34-46 %]
03/22/2011 29.1 %
[34-46 %]
and the highly variable PLT, as you predicted:
03/27/2011 144 K/mcL
[160-400 K/mcL]
03/23/2011 114 K/mcL
[160-400 K/mcL]
03/22/2011 174 K/mcL
[160-400 K/mcL]
LikeLike
Yeah, I was also confused about why the color and structure of her sweaters (or are they shirts?) was important.
LikeLike
Yay for our tax dollars funding the FDA! Are there issues with the FDA? Sure. Do they force snake oil vendors to prove their claims? Yes. Is this a good use of taxpayer funds (as well as the user fees they charge to the industry) – very much so yes!
Sure it’s for drugs, but say what one will about the Hatch Waxman Act of 1984, but it seems to work ok to help balance the needs of the populace versus the needs of capitalists:
https://en.wikipedia.org/wiki/Drug_Price_Competition_and_Patent_Term_Restoration_Act
LikeLike
This would make sense if we could trust what machines Theranos is using to process the results. Everything is in question.
LikeLike
Hey, if they want an independent analysis of their testing data by an Epidemiologist then I am always looking for topics to write papers on. 🙂
LikeLike
Theranos shows charts of the correlations of their tests to reference methods on their site here: https://www.theranos.com/our-lab
6 of the charts have a single point on the high end of the scale that improves the correlation. That is probably part of the method – including data at the high end of the expected normal values. For example, the bilirubin data has a correlation of 0.96 if you don’t include that high point; including the high point, you get R squared = 0.97. But the effect of having a single point far from the values of the others – always improves R squared – even if that point is way off. On this chart, if that point was (50,100) instead of (1.9,1.9), R squared is 0.9996.
Also in the bilirubin chart, the granularity of the Theranos data is 0.1, but the granularity of the predicate data is 0.05. I wonder why Theranos doesn’t measure bilirubin at that granularity.
LikeLike
Interesting. I’m not familiar with typical correlations in this literature base, but when I see R-squares that large I always look to make sure something isn’t wrong or use a better association/effect statistic. It sounds like the FDA didn’t buy it. Theranos needs to pull back the curtain for their data to be trusted.
LikeLike
When I found out about this (a year or two ago), I decided that Theranos was probably fraudulent.
Note well, though: I haven’t bothered looking at their methods or evidence at all. I’m just applying the heuristic that if you’re willing to put Henry freaking Kissinger on your board, you’re evil, and if you’re evil, then you’re probably willing to lie.
(I should note I wouldn’t apply this heuristic to a company like Palantir. Their raison d’etre is to help the surveillance state use big data to spy on and abuse dissidents, and so putting war criminals on the board would be par for the course.)
LikeLike
Try this, you have a technology which sort of works but are having difficulty bringing up to the FDA standard, but it is a new technology. The funders are getting antsy. What to do, what to do.
LikeLike
Good news! One of Theranos’s 250 tests has been approved by the FDA! Now you can know if you have Herpes Simplex 1. Or, you could use this self-test – have you ever had a cold sore? Then you have Herpes Simplex 1, just like 85-90% of all adults.
LikeLike
Ha.
LikeLike
The board composition is not “fancy” . It is made up solely of people she needs for CONTACTS. She chose her board to exert INFLUENCE upon outside entities ( government, military, etc). It suggests the company head is insecure about her technology, and needs the reassurance of old, connections-savvy people, instead of people foccussed on the inner workings of her tech.
LikeLike