3

Probabilistic Genotyping

This workshop session focused on probabilistic genotyping (PG; see Box 3-1) and featured presentations from panelists with expertise in forensic analysis, genetics, computer science, and law. It was moderated by Alicia Carriquiry, Iowa State University. Panelists called attention to various aspects of the use of PG in law enforcement, including its processes, benefits, challenges, and the need for standards and regulations.

OVERVIEW OF PG

Craig O’Connor, New York City Office of the Chief Medical Examiner, provided a brief overview of PG for workshop attendees and participants. He explained that PG helps police, laboratory, and forensic professionals compare DNA found at crime scenes with that of potential suspects, even when the DNA is limited in quality, damaged, or mixed from different people. Todd Bille, National Laboratory Center of the U.S. Bureau of Alcohol, Tobacco, Firearms and Explosives, explained that PG is utilized broadly to obtain the most information possible from a DNA profile. He stated that PG involves the multidisciplinary use of biological modeling, statistical theory, computer algorithms, and probability distributions to analyze forensic DNA samples. PG combines these tools to calculate likeli-

hood ratios¹ and/or infer genotypes for the DNA typing results of forensic samples. Bille pointed to the Scientific Working Group on DNA Analysis Methods (SWGDAM, 2023) Guidelines for the Validation of Probabilistic Genotyping Systems, which include a formal definition of PG and a framework for forensic laboratories to validate PG (see Box 3-2). PG has two main functions, he explained:

1. mixture deconvolution, which results in a list of possible contributing genotypes and their associated weights or probabilities, and
2. calculating the statistical weight of a sample compared with the DNA of a person of interest, based on relevant population databases.

The results of PG could indicate a very high likelihood ratio for a person who is included in a forensic sample, Bille said. He explained that PG also has the ability to discriminate between true contributors and noncontributors to a mixed DNA profile. PG makes more efficient use of the data present in the DNA profile and can potentially provide evidence of exclusion for individuals who may not have been excluded with previous technologies. PG is less subjective than other technologies, which, argued Bille, allows for greater consistency over time within the laboratory and between analysts within the lab.

The forensic community is quickly adopting PG, with the number of laboratories using PG increasing from around 10 in 2016 to almost 90 in 2023 (Figure 3-1). Bille noted that training is essential for the proper use of PG because the software is not automatically configured for use and requires individuals with knowledge and skill to properly implement and maintain. Analysts need to be trained in DNA interpretation and manual deconvolution of mixtures so that they can compare the outputs of PG with what would logically be expected based on validated parameters. If the output from the software does not match what the analyst would expect, the analyst should be able to detect the problem and identify what went wrong, said Bille. If the software models the DNA profile incorrectly (e.g., produces a list of possible genotypes and associated weights that do not properly reflect the DNA profile content) and this goes unnoticed by the analyst, inaccurate conclusions may be reported.

___________________

¹ Likelihood ratios are statistical assessments of the strength of evidence given two potential scenarios or propositions. For example, in a criminal case involving a person of interest, a forensic laboratory may assess the strength of evidence that a person of interest did contribute to a DNA sample found at a crime scene, compared with the strength of evidence that they did not contribute to that sample. PG thus does not result in a binary yes/no inclusion or exclusion of an individual from a forensic DNA sample; instead, it results in a likelihood ratio that must be interpreted by the forensic analyst.

Speakers explained that PG software creates detailed mathematical models that capture the characteristics of the DNA data obtained from the laboratory, such as peak heights, stutter patterns, and other artifacts. This modeling process is specific to each lab’s procedures and instrumentation. The software quickly and simultaneously considers various factors, such as potential contributors, DNA template amounts, degradation, inhibition, and stochastic effects. After modeling the DNA data and accounting for the various factors, the software calculates likelihood ratios that provide a statistical assessment of the weight of the DNA evidence under different propositions (e.g., whether a person of interest contributed to the DNA mixture).

CONSIDERATIONS FOR PG USE BY LAW ENFORCEMENT

Opportunities

Considering what makes PG different from other advanced forensic DNA analysis tools, Bille noted that using PG can enable greater differentiation between DNA profiles (see Box 3-3). This can allow for the exclusion of people from a forensic sample who may not have been excluded with previous, less precise, technologies. Bille also noted that PG is less subjective than other forensic DNA analysis tools, which allows for the potential of greater consistency over time from individual analysts and across laboratories generally.

Rebecca Wexler, University of California, Berkeley, stated that PG could be helpful in solving crimes. She pointed out that PG can provide more information for juries to evaluate the totality of evidence in a case. Wexler also stressed the need for proper training and preparation for attorneys to accurately present statistical evidence in court and noted that the implementation of PG is not without its challenges and ethical considerations.

Challenges and Ethical Considerations

Understanding and Education

Discussing the risks around use of PG, Norah Rudin, Forensic DNA Consulting, explained that PG requires more understanding, education, and training than other methods for estimating the statistical weight of evidence

for forensic DNA profiles. Understanding the results is a struggle not just for laypersons or attorneys, but also for analysts. Describing the results of PG requires using very specific language, and if the language is not used correctly, it can introduce subtle errors and biases. In addition, PG cannot overcome sample and profile limitations—a sample that is of poor quality and/or quantity cannot necessarily be rescued even by advanced tools like PG. Another limitation of PG, Rudin mentioned, is that no true value exists upon which to calibrate systems. That is, an expected likelihood ratio (the final output of PG) cannot be predicted (although an upper limit can be determined based on a single source profile). These limitations, she said, are insufficiently appreciated. Offering conclusions reached using PG about samples relevant to a crime event in court gives a “patina of science” that may lack a full understanding and description of the limitations, including sources of uncertainty and variability.

Uncertainty and Variation

Rudin identified two sources of uncertainty and variation that are extrinsic to PG—technological and human.

Rudin explained that the procedure for performing forensic DNA typing requires making multiple copies of particular DNA fragments in a sample so that sufficient material is available to analyze. This process, called amplification, is conceptually similar to making photocopies of a document. However, Rudin specified that if the same DNA sample is amplified multiple times (say, five times), the analyst may end up with five different results. She explained that this difference occurs due to random variations (called stochastic sampling) that can occur both before and during the amplification process. These variations can cause significant differences in the results ultimately obtained via PG, sometimes even more so than the differences you might see when using different PG programs, said Rudin.

Human factors, such as choosing the analytical threshold, can also have a significant impact on the results, continued Rudin. She explained that analytical thresholds can be based on validation studies but are generally implemented via policies that lack adequate accountability mechanisms. Importantly, Rudin noted, an analytical threshold is not selected or modeled by the software—rather, the threshold is determined by the laboratory or analyst based on lab protocols and policies before entering data into PG. The variance in analytical thresholds can affect the apparent number of contributors to the sample, as well as perceived missing information, and can affect the final PG result. Regardless of a technology’s sophistication and power, said Rudin, “if you don’t ask the right question, you won’t get the right answer.” The hypotheses that are posed are generally chosen by the laboratory, and the choice of those hypotheses is a significant factor in determining the final PG results. For example, are there assumed contributors to the sample, are there potentially concurrent contributors, and are there relatives involved? These factors can complicate the analysis and the hypotheses that are compared, she said, and can result in significantly different results. Sarah Chu, Perlmutter Center for Legal Justice at Cardozo Law, added that reducing variation and setting a standard of practice requires oversight and accountability systems. Jeanna Matthews, Clarkson University, pointed out that reducing variation is not the same thing as improving accuracy; variation can lead us to investigate why results differ and to seek out the correct answer.

Software Verification and Validation

Turning to considerations for implementation on the technological side of PG, Matthews reminded the audience that all computer software—in-

cluding PG—has bugs. She emphasized that there are well-established best practices for verification and validation of software, and expressed concern that PG is not currently tested using these practices. Matthews explained that verification refers to whether the product was built correctly using the correct process, and validation refers to whether it was the right product that was built—that is, does the output meet the needs for which it was built? The well-established practices for building reliable software and iteratively identifying and removing defects include requirements definition, design definition, unit testing, integration testing, acceptance testing, regression testing, bug/defect tracking, version control, debuggers, and code coverage tools. Matthews noted that the U.S. government uses the IEEE (2017) Standard 1012 for System, Software, and Hardware Verification and Validation for critical software throughout the government, including software for aviation, at the National Aeronautics and Space Administration, and at the Nuclear Regulatory Commission (Box 3-4). The level of effort put into verification and validation, said Matthews, is guided by the integrity and severity level; for example, if the software impacts human life or liberty, the verification and validation process should be conducted at the highest level.

Ethics and Legality

Panelists also discussed ethical and legal concerns regarding the use of PG software by law enforcement. In this vein, Rudin noted that commercial PG programs present challenges to fair and equal access. Because the software itself is expensive and proprietary, it has not been possible to conduct true independent testing. Furthermore, she explained, independent experts face significant challenges in accessing the software to review casework, especially if the manufacturer considers them competitors. And, said Rudin, because people of color are overrepresented in the criminal justice system, they are disproportionately impacted by these limitations in access to commercial PG programs. She emphasized the need to identify options to provide equitable access for both research and casework.

Wexler agreed that lack of access is a major issue with PG. Specifically, she outlined how some software developers are currently able and incentivized to withhold relevant information entirely in a criminal case by citing trade secrecy protections. Wexler argued that instead of blocking defense access to PG entirely, trade secret privilege should merely provide a right to a reasonable protective order and, potentially, to courtroom closure. She noted that the United States is a “model to the world” in its commitment to fair and open criminal proceedings, which includes the right for the accused to confront witnesses against them and the obligation of prosecutors to disclose evidence to the defense. Wexler suggested that like all other

evidence, evidence from PG systems must be subject to robust adversarial scrutiny, and trade secrets should not be an exception to that requirement.

Considerations for Implementation

Expanding on her earlier discussion of trade secrets and protected information, Wexler pointed to the Justice in Forensics Algorithms Act (2024), a bill that has been introduced several times, and contains model language that would prevent trade secrets from being used to impede discovery of evidence: “There shall be no trade secret evidentiary privilege to withhold relevant evidence in criminal proceedings in the United States courts” (§ 2[b][1]). This language, said Wexler, could be used in a best practices guideline or as a requirement for procurement or federal grants. Inaccuracy and obstacles to transparency are an equity issue, said Wexler, due to the unfortunate reality that those accused of and victimized by crime

are disproportionately from poor communities and communities of color. Wexler noted that the criminal legal system relies conceptually on equally matched adversaries with a neutral fact finder, but when there are unnecessary impediments to one side doing their job, it skews the ability of the system to reach accurate outcomes.

Wexler echoed the conclusions of the National Institute of Standards and Technology and the Government Accountability Office that PG should be subject to review by independent researchers who have no stake in the outcome. Unfortunately, she noted, many vendors refuse to allow this type of access. Wexler shared that she and her colleagues in computer science, law, and forensic genealogy put together a team to conduct a review of PG. However, when they reached out to companies to purchase a research license, they were ultimately denied, with one reporting that they do not provide research licenses. Rudin said that she is “mystified to some extent” by the reluctance of developers to have independent groups confirm the

software. For programs that are, in fact, accurate and reliable, independent confirmation would reassure all interested parties.

Under the Daubert standard, which is used to determine admissibility of evidence (Daubert v. Merrell Dow Pharmaceuticals, Inc., 1993), judges consider whether the method used to generate scientific evidence has been subjected to publication and peer review and accepted within the relevant scientific community, said Wexler. Representatives of PG companies have testified under oath that their product is subject to peer review, she said, and that the results should therefore be admissible. However, they do not allow independent research into quality assurance and validation. Matthews added that the “relevant scientific community” does not usually include software engineering, despite the fact that the evidence in question was generated by software.

Bille highlighted the importance of establishing standards and regulations for PG. He mentioned the need for validation of PG software to ensure its reliability and accuracy by the laboratories. Bille also discussed the role of organizations such as SWGDAM, Organization of Scientific Area Committees, the UK Forensic Science Regulator, and the DNA Commission of the International Society for Forensic Genetics in developing standards for the validation and use of PG. In addition, forensic laboratories that participate in the Combined DNA Index System (CODIS) must comply with the National Quality Assurance Standards and accredited labs must comply with the standards of their accrediting body such as the American National Standards Institute’s National Accreditation Board. Wexler asserted that vendors of PG should be required to allow independent audits, and procurement guidelines for law enforcement should include a requirement that the tools are subject to independent peer review from people with no stake in the outcome. Matthews emphasized that it is not “too late to get this right in the DNA space.” Software will continue to be developed and implemented within the criminal legal system, and she suggested that rather than continue operating without consistent standards, the field should require that software engineering best practices be followed and push back against trade secrets protections being used as a shield to refuse disclosure of evidence. Public trust requires transparency and accountability, said Matthews, and we need to make the necessary changes now to set a course for future technologies.

REFLECTIONS

Following panelist presentations, Chu, serving as discussant, offered her reflections and perspective on the topics discussed. When forensic DNA analysis was first developed, she noted, there were many difficult conversa-

tions about the associated ethical, legal, social, and justice issues. As DNA analysis become more ubiquitous and less expensive, these conversations, questions, and critiques largely fell away. Now there is an explosion in technologies, she said, that has left the public and decision-makers struggling to catch up with ethical, social, and justice implications. Chu suggested that while PG has potential to assist in criminal investigations, there is a need to carefully consider these implications and to ensure that the technology is reliable and useful. Getting it right is important for everyone’s sake, she said, but particularly for historically overpoliced communities of color, where the harms of these new technologies are concentrated.

“What do we owe each other when we are implementing PG systems?” posed Chu. She suggested that justice and transparency require robust and comprehensive validation studies and standards that are not restricted by commercial restraints or adversarial legal systems, as well as use of PG that is data driven rather than politically driven. Validation is integral to generating reliable results, the conclusions we can draw from the results, and how much confidence we have in the results. Chu emphasized the need for federal support to help laboratories conduct validation studies and to make the data publicly available, noting that most labs lack the infrastructure and resources to maintain databases to share information. Centralized databases supported by the federal government could thus play a key role in improving transparency. Chu identified another important role for the federal government—as well as state and local entities—in conditioning grants and procurement on requirements that software be developed under IEEE Standard 1012 (2017) and that software be made accessible and available to both researchers and defendants. Furthermore, labs should disclose their PG records so that defense attorneys can review the assumptions and choices that underlie the results presented in court. If we permit technology companies to use trade secrets arguments to avoid sharing PG evidence, said Chu, there is a risk that as the role of technologies in the criminal legal system expands, more and more evidence will be withheld. This would make it impossible to have a “fair and just system,” she said.

Finally, Chu suggested, decisions about the use of forensic DNA technologies need to be driven by data rather than politics. As an example of this problem, Chu said that laboratories across the country have been told by law enforcement and political leaders that they should swab every gun for genetic material. These swabs tend to produce low quantity, high contributor mixtures which are among the hardest samples to interpret through PG. Limited law enforcement resources, she suggested, could instead be used to process higher-yield cases and use sexual assault kits, for example. In addition, results from gun swabs generally cannot be entered into CODIS, so these tests incentivize the growth of local DNA databases

that are not required to follow state regulations. Testing policies should be determined in conjunction with forensic scientists and other stakeholders, said Chu, calling for open, transparent discussions among experts and members of the public about DNA collection, retention, and expungement policies. Chu suggested that taking these steps will help ensure that law enforcement does not become overly reliant on technology and the type of DNA collection practices that can erode trust in law enforcement.