Laypersons rival popular software in ability to predict recidivism, study says
Shutterstock.com.
Software that predicts the risk of a defendant committing new crimes performed no better than people responding to an online survey, a Dartmouth College study has found.
The study found that the online group, who had no presumed criminal justice experience, had an accuracy rate of 67 percent in predicting recidivism, compared 65.2 percent for the software known as COMPAS (Correctional Offender Management Profiling for Alternative Sanctions).
The software uses 137 pieces of information to make a prediction. But the study, published in Science Advances, found the same level of accuracy can be achieved using only two variables: a defendant’s age and number of prior convictions.
COMPAS has been used to assess over 1 million offenders since its development in 1998. Beginning in 2000, the software added a recidivism component that predicts a defendant’s risk of committing a misdemeanor or felony within the next two years.
Such software has been used in bail, parole, and sentencing decisions, spurring objections from defense lawyers who raise due process concerns about its proprietary algorithm.
The online study participants saw a short description of a defendant that included sex, age and previous criminal history, but not race. The participants were asked whether the defendant would commit another crime within two years of their most recent crime.
The researchers used 1,000 defendant descriptions that were divided into 20 equal subsets. Each person in a group of 20 study participants saw the same subset. The researchers used the prediction reached by a majority of each 20-person group.
The accuracy rate for black defendants was 68.2 percent for the layperson group, compared to 64.9 percent for COMPAS, while the accuracy rate for white defendants was 67.6 percent for the laypeople, compared to 65.7 percent for COMPAS.
But the study found that the mistakes affected blacks and whites differently. The rate at which defendants were wrongly predicted to reoffend—a false positive—was 37.1 for black defendants and 27.2 percent for white defendants in the laypeople study. The false positive rate for COMPAS was 40.4 percent for black defendants and 25.4 percent for white defendants.
Similarly, the false negative rate for black defendants was 29.2 percent compared with 40.3 percent for white defendants in the laypeople study. The false negative rate for COMPAS was 30.9 percent for black defendants and 47.9 percent for white defendants.
The study was repeated with a second group of 400 participants, only this time the defendants’ race was included. The accuracy rate was not significantly different than in the first group.
The new study was conducted by Dartmouth computer science professor Hany Farid with student Julia Dressel.
“The entire use of recidivism prediction instruments in courtrooms should be called into question,” Dressel said in the press release. “Along with previous work on the fairness of criminal justice algorithms, these combined results cast significant doubt on the entire effort of predicting recidivism.”
