Defense lawyers want to peek behind the curtain of probabilistic genotyping
Science under scrutiny
Built on biology, computer science and statistics, the world of probabilistic genotyping is niche. With few people who can understand these tools, the challenge of analyzing them is compounded by the fact that companies developing these tools “black-box” the computer code. This means there is no or limited capability to review the math; therefore, it cannot be independently challenged.
Richard Torres, a staff attorney in the DNA unit of the New York Legal Aid Society in New York City, and scientists such as Krane are concerned about the lack of transparency.
“My biggest issue is with access to the source code,” Torres says. “It’s a confrontation issue,” a reference to the Sixth Amendment right to confront a witness who makes a claim against a defendant. Torres argues that an algorithm behind a genotyping tool is speech that makes a claim against a defendant—so the defense has a right to confront and question the algorithm, not just the scientist who made it.
Organizations like Cybergenetics and the Institute of Environmental Science and Research provide defense counsels access to the tool and supplemental materials such as validation studies, but the information is incomplete to protect the company’s intellectual property. Less frequently used tools, including Lab Retriever, LRMix and LikeLTD, have their source code available for download without limitations.
Defense access to the source code has been litigated around the country with mixed results.
In California, People v. Chubbs, a cold murder case from the 1970s, brought this issue to light in 2012. The prosecution used evidence from TrueAllele, saying DNA found on the victim and Martell Chubbs’ sample was “1:62 quintillion times more probable than a coincidental match to an unrelated black person.” The trial court determined that Chubbs was entitled to examine the source code under protective order.
On appeal, this decision was overturned in 2015. The appeals court said that Chubbs’ stated reasons to access the source code, even under protective order, did not outweigh trade secret protections. Further, as the court writes, “access to TrueAllele’s source code is not necessary to judge the software’s reliability” because validation studies and expert testimony are sufficient to make that determination.
This view is shared by Mark Perlin, CEO of Cybergenetics, the company at issue in the Chubbs case. He thinks that full algorithmic transparency is not necessary, and that the scientific process can and should police forensic science.
The challenge here, says Erin Murphy, a law professor at New York University and author of Inside the Cell: the Dark Side of Forensic DNA, is that genotyping software validation studies are done almost entirely in-house by the company itself, putting their validity into question. She says that “one of the key ways to understand the strengths and weaknesses of these programs is to make them really transparent.”
Siding with Murphy, Judge Valerie Caproni of the U.S. District Court for the Southern District of New York determined in Johnson v. U.S. in July 2016 that the Forensic Statistical Tool’s “source code is ‘relevant [and] admissible,’ ” at least during a Daubert hearing, a pretrial hearing where the admissibility of expert testimony is challenged. Caproni provided a protective order, which she later lifted when journalists at ProPublica filed a motion arguing that there was a public interest in the code. The source code is now available online.
With this debate ongoing, requiring access to source code may be insufficient, says Nathan Adams, a systems engineer at Forensic Bioinformatics, a Fairborn, Ohio-based consultancy led by Krane.
“Even if it’s an open sourced program … and there is interest” in doing an evaluation, there are resource and ability limitations, says Adams. “The field of forensic DNA hasn’t emphasized the skill set of software quality assurance,” he adds.
Frank Pasquale, a law professor at the University of Maryland and author of The Black Box Society, agrees with Adams, saying it is a “huge problem” to find experts in computer science, biology and law. “Before we mass-deploy these things, we have to make sure these experts are equally spread out,” he says.
Without any indication that the proliferation of these tools will wait for a cadre of experts to be created and deployed, courts and scientists will continue to grapple with questions using the resources they have.
Acknowledging this debate, Rhonda Roby, the supervising DNA criminalist at the Alameda County Sheriff’s Office forensic biology unit in California, strikes a balanced tone. “I think the bottom line is that it’s a critical time, right now, to closely evaluate and not dismiss the technology,” she says. “It’s that good that we need to consider it.”
This article was published in the December 2017 issue of the
