Home > Uncategorized > Sketchy genetic algorithms are the worst

Sketchy genetic algorithms are the worst

June 2, 2016

Math is intimidating. People who meet me and learn that I have a Ph.D. in math often say, “Oh, I suck at math.” It’s usually half hostile, because they’re not proud of this fact, and half hopeful, like they want to believe I must be some kind of magician if I’m good at it, and I might share my secrets.

Then there’s medical science, specifically anything around DNA or genetics. It’s got its own brand of whiz-bang magic and intimidation. I know because, in this case, I’m the one who is no expert, and I kind of want anything at all to be true of a “DNA test.” (You can figure out everything that might go wrong and fix it in advance? Awesome!)

If you combine those two things, you’ve basically floored almost the entire population. They remove themselves from even the possibility of critique.

That’s not always a good thing, especially when the object under discussion is an opaque and possibly inaccurate “genetic algorithm” that is in widespread use to help people make important decisions. Today I’ve got two examples of this kind of thing.

DNA Forensics Tests

The first example is something I mentioned a while ago, which was written up beautifully in the Atlantic by Matthew Shaer. Namely, DNA forensics.

There seem to be two problems in this realm. First, there’s a problem around contamination. The tests have gotten so sensitive that it’s increasingly difficult to know if the DNA being tested comes from the victim of a crime, the perpetrator of the crime, the forensics person who collected the sample, or some random dude who accidentally breathed in the same room three weeks ago. I am only slightly exaggerating.

Second, there’s a problem with consistency. People claiming to know how to perform such tests get very different results from other people claiming to know how to perform them. Here’s a quote from the article that sums up the issue:

“Ironically, you have a technology that was meant to help eliminate subjectivity in forensics,” Erin Murphy, a law professor at NYU, told me recently. “But when you start to drill down deeper into the way crime laboratories operate today, you see that the subjectivity is still there: Standards vary, training levels vary, quality varies.”

Yet, the results are being used to put people away. In fact jurors are extremely convinced by DNA evidence. From the article:

A researcher in Australia recently found that sexual-assault cases involving DNA evidence there were twice as likely to reach trial and 33 times as likely to result in a guilty verdict; homicide cases were 14 times as likely to reach trial and 23 times as likely to end in a guilty verdict.

Opioid Addiction Risk

The second example I have today comes from genetic testing of “opioid addiction risk.” It was written up in Medpage Today by Kristina Fiore, and I’m pretty sure someone sent it to me but I can’t figure out who (please comment!).

The article discusses two new genetic tests, created by companies Proove and Canterbury, which claim to accurately assess a person’s risk of becoming addicted to pain killers.

They don’t make their accuracy claims transparent (93% for Proove), and scientists not involved with the companies peddling the algorithms are skeptical for all sorts of reasonable reasons, including historical difficulty reproducing results like this.

Yet they are still being marketed as a way of saving money on worker’s comp systems, for example. So in other words, people in pain who are rated “high risk” might be denied pain meds through such a test that has no scientific backing but sounds convincing.

Enough with this intimidation. We need new standards of evidence before we let people wield scientific tools against people.

Categories: Uncategorized
  1. Matthew Thomas
    June 2, 2016 at 9:53 am

    I thought this post was going to be about genetic algorithms! https://en.wikipedia.org/wiki/Genetic_algorithm

    I would have agreed with the title!

    Liked by 1 person

    • June 2, 2016 at 11:11 am

      I thought the same, unless Cathy intended that as a pun.


    • RTG
      June 2, 2016 at 6:08 pm

      Yeah, this is a little confusing. Maybe she does mean using “genetic algorithms” to analyze genetics?


  2. Mike
    June 2, 2016 at 10:53 am

    Your last paragraph is the killer. The rules of evidence in law–that an ethical lawyer must cherry-pick supporting results as it is unethical to undermine their own case–are deeply unethical in the sciences. I don’t see how the lawyers will move toward the scientific ideal to be ruthlessly honest about what you know and don’t know. And even if the lawyers wanted to, it’d make each case a lot more work and so we’d have to dramatically reduce the number of people we tried to lock up and then accept a much higher rate of acquittals for those that are tried. Does this sound like the direction the US is heading?


  3. Klondike Jack
    June 2, 2016 at 1:05 pm

    Sorry, but there’s no profits to be made or market share to be gained via new standards of evidence. Unsubstantiated claims of utility and efficacy buttressed by plausible deniability are the only way to go when Gresham’s Dynamic controls the bottom line.


  4. Min
    June 2, 2016 at 2:37 pm

    I remember when, during the OJ criminal trial, the head of Cellmark (if I remember the name correctly), which did the DNA testing, stated that they no longer did quality testing by having samples of known DNA submitted to them in the blind by independent testers. I was flabbergasted! How could Cellmark’s test results be trusted?


  5. RTG
    June 2, 2016 at 6:14 pm

    I’m surprised, though probably shouldn’t be, that the Opioid Addiction Risk tests are allowed to be used for decision-making. Wouldn’t they need to be FDA approved? Or am I hopelessly naive?

    My only direct experience with DNA analysis is with the cell-free prenatal fetal DNA tests. It seems almost like science fiction, but they can draw blood from my arm which apparently contains traces of my fetuses blood…and those traces are enough to provide fairly accurate screening tests for a handful of trisomies (T21-99% accurate, T13 and T18 ~ 90% accurate, XY trisomies or monosomies and sex identification – 95%) Numbers may be slightly off, but these are around what I recall. I’ve seen several studies since these tests were approved in 2011 suggesting that the accuracy continues to be correct even with widespread use. My understanding, though, is that these tests were available for years before they got FDA approval…and the likelihood is that they can do more than what they are currently approved to do.

    It seems crazy that you might deny an injured person pain killers based on a risk assessment that is only 93% accurate.


  6. Guest2
    June 2, 2016 at 10:28 pm

    I think Opioid Addiction Risk testing can be seen as an example of the cultural trend to “reduce society to the smallest rational units” possible, in this case to DNA. The goal — and the promise — of such rationalization is control and systematization, in this case, of behavior.


  1. June 3, 2016 at 6:57 am
Comments are closed.
%d bloggers like this: