Predictive risk models for prisoners with mental disorders
My friend Jordan Ellenberg sent me an article yesterday entitled Coin-flip judgement of psychopathic prisoners’ risk.
It was written by Seena Fazel, a researcher at the department of psychiatry at Oxford, and it concerns his research into the currently used predictive risk models for violence, repeat offense, and the like, which are supposedly tailored to people who have mental disorders like psychopathy.
Turns out there are a lot of these models, and they’re in use today in a bunch of countries. I did not know that. And they’re not just being used as extra, “good to know” information, but rather as a tool to assess important decisions for the prisoner. From the article:
Many US states use such tools to assess sexual offending risk and to help decide whether to exercise their powers to detain sexual offenders indefinitely after a prison term ends.
In England and Wales, these tools are part of the admission criteria for centres that treat people with dangerous and severe personality disorders. Outside North America, Europe and Australasia, similar approaches are increasingly popular, particularly in clinical settings, and there has been a steady growth of research from middle-income countries, such as China, documenting their use.
Also turns out, according to a meta-analysis done by Fazel, that these models don’t work very well, especially for the highest risk most violent population. And what’s super troubling is, as Fazel says, “In practice, the high false-positive rate probably means that some offenders spend longer in prison and secure hospital than their true risk would suggest.”
Talk about creepy.
This seems to be yet another example of a mathematical obfuscation and intimidation that gives people a false sense of having a good tool at hand. From the article:
Of course, sensible clinicians and judges take into account factors other than the findings of these instruments, but their misuse does complicate the picture. Some have argued that the veneer of scientific respectability surrounding such methods may lead to over-reliance on their findings, and that their complexity is difficult for the courts. Beyond concerns about public protection, liberty and costs of extended detention, there are worries that associated training and administration may divert resources from treatment.
The solution? Get people to acknowledge that the tools suck, and have a more transparent method of evaluating them. In this case, according to Fazel, it’s the researchers who are over-estimating the power of their models. But especially where it involves incarceration and the law, we have to maintain an adherence to a behavior-based methodology. It doesn’t make sense to put people in jail an extra 10 years because a crappy model said so.
This is a case, in my opinion, for an open model with a closed black box data set. The data itself is extremely sensitive and protected, but the model itself should be scrutinized.
mathbabe 
This raises a question, one that you have probably already addressed on this blog: how do you convince people to investigate in a detached, rational manner an issue that, by its nature, raises strong emotional responses. I think the general public will have much less sympathy for a bad model such as this one, if the consequence of that model is that sexual offenders are kept in prison. How do you keep focus on the “badness” of the model and ways to improve the model, rather than on the emotional response to the behavior that is being modeled? Surely this same question arises in models of disease, drug testing, etc.
LikeLike
So long as the models lead to false positives, i.e., keep people in jail, then the politicicans will not likely act, since that can be presented as “getting soft on crime”; and the prision industry will make sure that the truth never gets wide audience.
The real problem is to stop creating businesses that use statistics to “predict” human behavior. While some examples of usefulness may exist, the temptation to abuse the technology is too great a threat to our civil liberties.
LikeLike
Cathy (and others),
You might be interested in reading this related paper by statistician-sociologist Richard Berk:
Click to access CompCrim%20copy.pdf
LikeLike
So I actually served on a jury that considered the kind of case you’re talking about.
Guy had served jail time for multiple sexual assaults. The laws of the Commonwealth of Massachusetts say that he “may be confined indefinitely for treatment after the termination of the person’s criminal sentence if the person is found to be a Sexually Dangerous Person in accordance with statutory procedures.” He has the right to ask a jury to declare him no longer a SDP something like once every year or two.
One of the key pieces of evidence presented by the Commonwealth was the STATIC-99, the sort of statistical tool addressed by Jordan’s article. Offenders are scored on ten items (mostly binary), then their total score is looked up on a table to find a putative probability of sexual recidivism over the next 10 years. The most heated confrontation at the trial was where a lawyer was trying (unsuccessfully) to yell a witness into agreeing that one of the STATIC points against the offender was unclear from the record and so maybe he was actually in the group with recidivism probability 14%, instead of the 31% group that the Commonwealth presented. (It’s been many years, so my numbers might be a little off.)
Jury deliberations were really interesting. The trial record had done a pretty good job at explaining how STATIC worked, and in particular we knew how little it actually took into account. Our discussions basically came down to: Okay, let’s assume that something like 30%-or-maybe-it’s-only-15% of the people kind of like our guy go on to commit another sex offense. We’ve heard a bunch of stuff about him that’s not what the STATIC cares about. Do those things make us feel like he’s at on the more-safe or more-risky side of his STATIC bucket? Answer: Oh yeah, on the more-risky side, for a bunch of reasons. He remained an SDP and “confined for treatment”.
As a mathematician, I thought the jury’s consideration of the statistics alongside all the other evidence presented was actually pretty reasonable.
LikeLike
I don’t think the causation necessarily works in the direction implied (poor models leading to unnecessary imprisonment). For example, way back when I was clerking after law school about five years ago, I was talking to someone who had done some work on a few of the civil commitment hearings that lead to the effective lifetime commitment to an asylum (after the offender had completed their prison term I should add). He basically told me that it seemed pretty clear that the actual rates of recidivism really didn’t jibe with the frequency of psychiatric testimony that the offender was too dangerous to let into society. In other words, this has been known by people involved in the judicial process for a while. However, the civil commitment hearing is before a jury, the fact of the sexual offense has been established as fact at the prior criminal trial, and it requires only a preponderance of the evidence weigh in favor of believing the pro forma psychiatric testimony, so its kind of a guaranteed verdict.
But the political history of these commitment hearings is fairly recent; they began to be used as an adjunct to criminal penalties about 10-15 years ago as a response to the demand to stop the scourge of sex offenders that had preceded unabated despite the heroic effects of sex offender registries and various grand guignol crime dramas.
So, the cynical interpretation is that the process was initiated as a way to punish sex offenders further and the rationale of danger to the public is an after the fact basis for doing so. A less cynical interpretation is that society wants sex offenders to be punished severely, so it will find a way to do so and justify it afterwards. If the models are discredited, then the process will evolve to punish without them unless society’s opinion (i.e., conventional wisdom) is changed along with the models.
(As a sidenote my outrage is kind of pedantic, as its mostly rule of law based. I would probably not be quite so upset if they just amended the sentencing laws to allow life imprisonment for the crimes. Its probably not a good idea, but its no worse than many of our other draconian criminal justice approaches)
LikeLike