6

Conclusions and Recommendations

BioWatch is an environmental monitoring system designed to detect genomic material from a set of aerosolized pathogens. It is owned by the Department of Homeland Security (DHS) but relies on a network of program users and partners in state and local jurisdictions as well as federal agencies. The core pathogen detection technology on which BioWatch relies is real-time polymerase chain reaction (PCR) and the laboratory analyses of BioWatch samples are conducted in state and local public health laboratories. To be effective, the program must operate as a public safety and health decision-making tool for federal agencies but, more importantly, for local system users. Responding to a detection that turned out to be false positive drains jurisdictional resources and could undermine confidence in the system when a true positive occurs. On the other hand, a false negative result could be a catastrophic error if a pathogen release was missed.

The BioWatch program drew on scientific knowledge (assays, detection methods, and pathogen genomic organization) that was available at the time of its inception. This base of knowledge has continued to evolve over the course of the program’s deployment. Multiple stakeholders, within and outside of DHS and the federal government, have subsequently put substantial effort into the composition of inclusivity, exclusivity, and background environmental panels for microbial detection assays and the development of assay performance and validation standards. In recent years, the BioWatch program also has supported work at Los Alamos National Laboratory (LANL) and Dugway Proving Ground

to assemble and qualify strain materials and conduct assay and operational testing of its system’s performance. These efforts and existing standards guidance (such as the Stakeholder Panel on Agent Detection Assays [SPADA], Public Health Actionable Assay [PHAA]/Federal Standards for Assay Performance and Equivalency [FSAPE], and from the Food and Drug Administration [FDA] and the Environmental Protection Agency, and others) provide a solid foundation for understanding assay performance and validation standards appropriate for the program’s mission.

FUNDAMENTAL PRINCIPLES OF PERFORMANCE STANDARDS AND VALIDATION

An assay performance standard describes requirements that must be met for the assay to be considered acceptable and describes how testing to validate this performance is to be carried out. Elucidation of performance standards that ensure that the BioWatch PCR assays have performance characteristics that provide sufficient confidence to program stakeholders is the motivating subject of this report. Although the report focuses on fundamental principles for the PCR assay component, it is important to emphasize that interpretation of an operational result (which is essential for user “actionability”) is a complex process involving multiple components of the system, including sample collection, sample extraction, PCR amplification, PCR detection, data analysis, and current law enforcement or intelligence agency threat assessments. Similarly, validation reflects not only specific testing conducted on the PCR assay’s performance, but an overall framework designed to ensure that the quality of the overall systems processes meet their intended purpose.

APPROACHES TO ASSAY PERFORMANCE CHARACTERIZATION AND VALIDATION

BioWatch currently operates using a two-stage testing process. Initial screening assays and reagents are provided through the Department of Defense (DOD) Critical Reagents Program (CRP) (testing for the presence of a single signature per pathogen). Subsequent secondary assays and reagents to verify the screening results are provided through the Centers for Disease Control and Prevention (CDC) Laboratory Response Network (LRN) (testing for the presence of several signatures per pathogen). A screening assay must be able to detect the presence of small amounts of nucleic acid from a tested pathogen (sensitivity) and minimize the rate of obtaining a false negative result. A secondary assay for verification or confirmation, on the other hand, needs to be able to precisely distinguish between the tested pathogen and closely related but nonpathogenic

organisms to minimize the rate of a false positive result being passed forward for action (specificity). The combination of the screening and secondary assays thus functions to provide an overall combination of high sensitivity and high specificity.

The BioWatch program operates under some historical assumptions and program constraints. These include using one screening assay per BioWatch threat organism, emphasizing stringent performance of the initial assay (as noted above, certainly such assays must not produce many false negatives), using a secondary PCR panel of 3-5 target signatures, and not incorporating additional rule-out assays for certain agents. These assumptions are challenged by the strain variability in agents of interest to BioWatch. Newly sequenced strains of target agents eventually will reveal potential false negative situations, and potential false positive situations from newly discovered near-neighbor genomes. Similarly, the program has focused its consideration of the development of multiplex assays primarily on the initial screen, which may reflect earlier days of multiplex PCR development, when successfully multiplexing more than a handful of assays was more difficult. Current PCR multiplexing, in solution or solid phase formats, is no longer subject to as many constraints, and BioWatch now can consider expanded multiplexed approaches that incorporate a larger number of assays to determine presence or absence of key threat agents.

A final assumption appears to be that the primary opportunity for significant assay performance testing against large inclusivity, exclusivity, and environmental panels is during laboratory validation. The biothreat agent releases the BioWatch system is designed to detect are rare events, against complex environmental backgrounds, and all assays have associated false positive and false negative rates. Using laboratory assay testing to obtain very high levels of statistical confidence that might be desirable for decision making (e.g., that an assay hypothetically be shown to have a false positive or false negative error rate of less than 1 in 100,000 tests with 95 percent confidence) is not practically achievable:

• The ongoing analysis of assay performance data from verification in user laboratories and from operational deployment of the assay in the program is critical.
• It will not be possible to undertake laboratory validation testing using sufficient replicates on enough different inclusivity, exclusivity, and environmental organisms in the presence of enough different types of potentially interfering substances and different filter conditions to characterize the long-term false positive and false negative rates of an assay to a level of statistical confidence users may desire operationally.

A process for testing assay performance to some reasonable level in the laboratory, balancing time, costs, and other constraints, is needed in concert with ongoing analysis of data obtained from assay verification in user laboratories and from operational deployment of the assay in the program. These additional sources of data will be important in providing further information on assay performance in jurisdictional settings and in obtaining a more accurate sense of assay false positive and false negative rates. This process will play a role in building confidence in the assay.

The use of rapidly expanding sequence databases can be of use to the program, and the increasing use of massively multiplexed assays also is changing the nature of some validation strategies. For example, when an individual assay in a multiplex is modified or replaced to reflect new knowledge of circulating strains, or a new assay is added to the multiplex, a subset of validation testing can be conducted rather than repeating the full range of performance data.

In addition to giving further consideration to the possibilities enabled by the use of multiplex assays, the report suggests several options that BioWatch could consider as alternatives to the model of initial screening and subsequent secondary assays and the associated processes of performance characterization and validation. These include

• Expanding the use of in silico screening during assay characterization and validation;
• Using a tiered or serial testing approach with certain performance characteristics validated more stringently than others based on the intended use of the assay;
• Developing subsets of assays for each of the tested BioWatch agents rather than a single set of screening and secondary assays (e.g., including additional types of rule-in and rule-out assays along with decision algorithms on when to use which tests); and
• Incorporating next-generation sequencing (NGS) into the program as technology and software development makes this increasingly feasible.

Recognizing the Work That Has Been Done

It is important to recognize the substantial body of work that the BioWatch program and its contractors have undertaken to assemble reference materials, design appropriate testing setups, provide guidance on performance standards, test and evaluate PCR assays, conduct testing to understand current system performance, and establish and operate a quality assurance (QA) program. A significant base of information char-

acterizing assay and system performance is available now to support the program’s successful operations.

A PCR Assay Standard to Meet BioWatch’s Mission

The committee focused on the BioWatch program’s mission in order to explore the technical considerations that inform design of appropriate PCR performance standards for the program and how they could be optimized. To be applicable to BioWatch, a PCR standard to characterize assay performance and conduct validation would need to meet the following basic goals:

• Establish that the assay has sufficient sensitivity to detect the release of a tested biothreat agent at a program-relevant concentration above the baseline of environmental background;
• Establish that the assay has sufficient specificity to detect tested pathogenic strains of concern but not to cross-react with non-tested strains and organisms within acceptable program false positive and false negative rates; and
• Be sufficiently robust for routine operational use by BioWatch jurisdictional laboratories and be otherwise acceptable to program users.

The committee’s conclusions and recommendations to address these goals are detailed below.

Testing Strategies: Opportunities to Expand in Silico Testing

The committee sees opportunities for BioWatch to increase its use of in silico testing. To make such testing successful, parallel efforts will be needed to ensure that reference databases contain as many high-quality sequences as possible (Chain et al., 2009) and are available to PCR assay and device developers through accessible reference databases. The availability of high-quality reference databases also will support PCR assay design by enabling signatures, primers, and probes to precisely target inclusivity strains while avoiding cross-reactivity with exclusivity strains (within the limits achievable given microbial diversity).

Recommendations

BioWatch should strive to test its PCR assays against as many known strains of its target agents and relevant near neighbors as possible. Undertaking all such testing through laboratory analysis alone is likely to be cost, time, and feasibility prohibi-

tive. Increased in silico testing should thus be incorporated into the program to support targeted laboratory testing.

To support this goal, the BioWatch program should work with other federal agencies and international partners to ensure that as many strains as possible are sequenced to at least a high-quality draft level and the data and associated reference materials made available to PCR assay and device developers through reference databases.

Assay Sensitivity and Specificity

Assay sensitivity relates to the ability of the assay to reliably detect the targeted organism at very low concentrations. Assay specificity is the ability to reliably detect all variants of the targeted organism without detecting nontargeted near-neighbor organisms. For pathogen detection assays such as those used by BioWatch, characterization of these parameters (whether defined in precisely this manner or defined under the overall probability-of-detection [POD] concept articulated by AOAC) provides important information for decision makers. The committee concluded the following:

• Either approach—using a predetermined acceptable minimum detectable level (AMDL), if there is stakeholder agreement on this value, or calculating minimum concentration of nucleic acid detectable by the assay with sufficient confidence, followed by stakeholder determination of whether this makes the assay acceptable for the mission—is reasonable to establish assay sensitivity as part of a performance standard.
• Using a reference strain approach to determining the limit of detection (LOD) and POD is reasonable given practical constraints on testing, but variation in true LOD/POD among strains is a potential issue of which programs making operational use of such assays should be aware and document. A reasonable option to help address this situation is to test a limited number of reference strains (such as n = 3-5) in a range of quantities near the expected LOD to get a sense of variation.

Similarly, the committee makes the following recommendations regarding inclusivity and exclusivity strains:

Recommendations

The panels of inclusivity and exclusivity strains specified for laboratory testing should seek to balance broad phylogenetic

and geographic diversity within the realm of practical testing. Inclusivity panel strains for laboratory testing should sample the genetic diversity represented by available strain collections with an emphasis on prioritizing those strains that cause significant morbidity and mortality, have high transmissibility, have a wide host range, or may be readily accessed by nefarious actors.

For laboratory testing of exclusivity panels (near neighbors and environmental organisms), a strategy of sample pooling is appropriate to increase testing efficiency. If a positive result is detected, each agent in the pool should be tested separately.

The PCR is Only One Step in the Process of a PCR Assay

The current study and resultant report have focused primarily on the core PCR reaction portion of a complete PCR assay, and the key processes before and after the PCR are not significantly addressed. PCR assay performance is critically impacted by the steps that lead into the assay and is influenced by the amplicon detection methodology and feedback from the steps that follow a result. Even a perfectly designed PCR with ideal specificity and sensitivity will fail with either a false positive or false negative, or an inconclusive, result if the sample collection, processing, purification, and reagent input are not optimized and standardized. In addition, detection methodology requires certain characteristics of the amplicon that results from the reaction. The ability to securely identify the target of interest can be affected by these interacting factors. Even the comprehensive work performed at LANL to establish standards, which was done with purified DNA samples, experienced issues with reagents. Similar rigorous and comprehensive studies should be performed, developed, and implemented for BioWatch in terms of sample processing and amplicon detection.

Recommendation

A PCR assay performance standard that meets program and user needs should include development and implementation of standards that encompass all of the individual steps in the assay from sample collection, extraction, amplification, detection, reagent quality control, and data analysis. There should be an integrated assay standard, as well, that includes all of the steps of a complete PCR assay as a unit.

Environmental Background

The BioWatch system is required to successfully detect a set of pathogenic microorganisms from a multitude of nonpathogenic or unrelated microorganisms present in the environment. This feature requires understanding how the assay result differs from a baseline. How to detect a true signal from background noise is complicated by the fact that the distribution of bacteria and viruses in different environments around the United States is not well characterized. The nucleic acid extracted from BioWatch collection filters must be amplified and detected from the complex background of environmental substances (e.g., soils) that may interfere with the assay. The committee drew the following conclusions:

• The information obtained from knowledge of environmental background could be used to prioritize the strains included in the environmental panel specified by the performance standard, in order to cover genetic diversity while keeping the panel size manageable. The information also could be used to improve the interpretation of assay results in the context of individual jurisdictions and minimize potential environmental nonthreat detections, and to provide a basis for improved understanding of assay predictive value in a particular jurisdiction or collector location.
• Expanded in silico testing is not likely to reduce the burden of laboratory testing against potentially interfering environmental substances. Conducting performance testing against a panel of such substances is useful to provide information on possible performance issues and to ensure sufficient assay performance in an operational context.

Recommendations

The Department of Homeland Security should undertake further characterization of environmental background organisms over time and at various locations. This sampling could build on the existing BioWatch network by using previously tested filters as samples and should also include judicious sampling of water and soil samples. Operational assay performance data in individual jurisdictions based on positive screening results might provide a source of invaluable data, as well.

Assay performance in the presence of known environmental contamination (e.g., rail dust in a subway system) should be addressed by assessing the assay detection limit for a limited number of inclusivity and exclusivity panel agents spiked on jurisdictional background filters.

A systematic process should be instituted to identify the root cause of positive assay results that do not appear to represent the targeted microorganism and to determine appropriate corrective actions to address the cause(s). This feedback and revision process should include the participation of laboratory experts in the jurisdictions and federal experts.

Reference Materials

A source of reference materials and reagents that is accessible to those who need it for development and validation is critical to the application of a performance standard. The DOD CRP currently serves this function for the screening assay and for quality control samples tested under the BioWatch QA program. The secondary assay and associated reagents are provided through the CDC LRN. The committee concluded that

• Maintaining and quality controlling reference materials (particularly materials derived from microbial strain cultures) are not trivial or inexpensive processes. Not all strains of a pathogen reported in the literature or identified clinically are easily accessible. Furthermore, the genomic organization of pathogens maintained in culture for extended time periods may no longer accurately reflect that of their pathogenic predecessors and some data simply are incorrect.
• The CRP credits its predicable and sustained funding through DOD as a necessary element for successful development and maintenance of its reference materials. If some or all of the materials needed for BioWatch assay performance testing and validation (now or in the future) will not be provided through CRP, a similar type of quality-controlled, sustainable, and accessible repository will need to be developed independently by DHS.

Recommendation

Federal agencies, including the Department of Homeland Security, the Department of Defense, the Department of Energy, the Department of Health and Human Services, and other partners, should collaborate to produce a reference document or database identifying which repositories contain strains or strain nucleic acids relevant for testing pathogen detection assays. Information on the requirements or procedures to obtain and use strains or strain materials (e.g., extracted nucleic acids) held by these repositories should be included, where possible. This information would provide a coherent source of information to

both federal agency programs and to individuals or companies involved in assay and device development and validation.

The Key Importance of Ongoing Assay and Reference Database Maintenance

Periodically re-examining performance is part of the ongoing process of QA. The performance of pathogen identification assays are inherently a process and not an end point: new strains and near neighbors will always be encountered, and the microorganisms themselves are constantly evolving. New sequence information continues to be obtained, which results in a dwindling number of potential unique PCR assays available for species/strain differentiation. A corollary is that for some agents of potential interest to BioWatch, it may not be possible to uniquely specify appropriate targets because of closely overlapping genomes among the target and nonpathogenic near neighbors. There already are signs of this with some strains of Bacillus cereus that are lethal to humans, very similar to Bacillus anthracis.

• BioWatch should be prepared to flexibly alter the PCR assays to accommodate these kinds of circumstances. If not, the next set of BioWatch assays is just moving the boundaries of the Maginot Line and will likely gradually but inexorably evolve toward obsolescence.
• The program could also consider including an additional, complementary agent test to achieve maximum sensitivity and specificity in its detection capabilities for certain agents, or for use in certain jurisdictions experiencing issues with a specific assay.

Similarly, documentation of the limits of the assay and well-described interpretation criteria provide important information so that users do not overextend their use of the data. A limitation of making knowledge actionable is not dependent on having perfect data, but on understanding the assumptions on and limitations of the data in order to effectively interpret results.

• When providing or deploying an assay, information on any potential issues with the assay (e.g., that strains x or y have been shown to cross-react or that the limit of detection is decreased to xx in the presence of contaminant yy) needs to be provided and built into operational guidance. Regularly updating the assay with this type of information as it becomes available may be easier than going back through full redesign and revalidation of the assay if

Recommendation

The BioWatch program should institute a process to regularly review and update all assays in light of new genetic knowledge. The program should re-evaluate its assays through at least annual in silico screening to take advantage of new genomic information in databases, update signatures, and identify potential false positive or false negative issues. Similarly, the composition of inclusivity, exclusivity, and environmental panels used for testing should be reviewed regularly and laboratory re-testing conducted as needed. Annual in silico and periodic laboratory re-testing of assay performance should be part of any assay performance standard accepted for BioWatch.

Starting Point for a BioWatch Assay Performance Standard

As discussed in the report, some reasonable compromise must be achieved in order to undertake performance characterization and validation work on assays deployed or being considered for deployment in the program. The committee has attempted to provide a starting point for a performance standards approach for conducting laboratory testing of assays that could provide reasonable statistical confidence for BioWatch users while seeking to keep numbers of tests and numbers of strains reasonable for intended use.

Recommendation

The following approach can serve as a starting point for a standard to provide confidence in PCR performance while seeking to undertake a reasonable amount of laboratory characterization and validation testing:

• A reasonable approach to determining an assay’s analytical LOD in a laboratory setting is to conduct serial dilution at a range of concentrations bracketing the estimated LOD, using n = 60 replicates with acceptance criteria of at least 58/60 for a given concentration, followed by appropriate curve fitting. LOD testing should also be conducted in the presence of realistic background matrix, such as previously tested BioWatch filters, to gain an understanding of how the LOD may be affected by operational background.
• The use of in silico screening presents a relatively fast and low-cost way to predict one part of assay performance using

• as many strains as are available as high-quality sequences in reference databases. In silico screening should be undertaken using the set of PHAA panels, if these strains’ sequences are available.

• The strain panels recommended by SPADA represent a good starting point for laboratory validation testing of BioWatch assays and were developed with stakeholder participation. However, many were developed in 2010-2011 and should be reviewed and updated to account for new genetic knowledge.
• The strategy used by LANL for testing detectability or nondetectability of inclusivity and exclusivity strains represents a reasonable model for BioWatch (testing n = 20 replicates, followed by testing 20 further replicates if an unexpected detect or nondetect result is obtained, with acceptance criteria of 20/20 or 39/40).

A ROBUST QUALITY ASSURANCE PROGRAM CONTRIBUTES TO CONFIDENCE IN THE SYSTEM BUT COMMUNICATION CHALLENGES REMAIN

Process, Transparency, and Communication

Various technical factors should be considered when establishing assay performance standards appropriate for BioWatch’s mission. The report discusses key principles and provides the committee’s conclusions and recommendations in these areas. Equally important, however, is the process that is used to establish such standards.

• Different performance standards have taken different approaches in developing their guidance. Although the BioWatch program is federally owned and relies on important federal relationships, it cannot succeed without close engagement with nonfederal partners. The use of subject-matter experts and inclusion of a stakeholder forum, of whatever type, in coming to agreement on the performance and validation standards used by the program will be key to ensuring that the standards are acceptable and meet program and user needs.

Since 2011, BioWatch has implemented a QA program that helps support the continuous process of validation, provides a mechanism to continue to build relationships with BioWatch jurisdictional laboratories, and provides a source of operational performance data that can be used by the program to improve the understanding and interpretation of assay

results. The BioWatch program also has made important strides since program inception in improving communication with state and local jurisdictions. Despite this progress,

• Jurisdictions, particularly jurisdictional laboratory personnel, indicate that they do not have access to the full range of assay performance validation and operational performance data they would like in order to support decision making based on assay results. This situation applies to both the initial screening assay and the secondary assay. This discrepancy limits the performance knowledge of those responsible for making response recommendations based on that very knowledge. As a result, there remains a lack of confidence in the assays from users in the jurisdictions. The committee does not lay blame on any one group for the communication deficiency. However, a system of effective data sharing and communication is a critical component of overall program success. There is an essential need for additional dialogue to implement a data-sharing and response system that meets program office, federal partner, and jurisdictional needs.
• Communication also may be useful in establishing which sample extraction and preparation methods work most effectively under which ranges of circumstances, and in helping to establish greater commonality in PCR input steps among jurisdictions.

Recommendation

Communication and information-sharing are necessary to establish confidence in a system. The BioWatch program, relevant federal agencies, and local and state jurisdictions should expand the communication and data-sharing that occurs among the network of federal and nonfederal partners involving both the screening and the secondary PCR assays. This effort would assist in establishing acceptable performance standards, enhance informed data interpretation and decision making, improve the ability to undertake root cause analysis of assay issues encountered by jurisdictions, and enable the collective identification and dissemination of actions as part of robust quality assurance.

STANDARDS APPROACHES FOR MULTIPLEXING REAL-TIME PCR ASSAYS

The committee concluded that existing standards approaches should be suitable for characterizing and validating the performance of multiplex PCR assays. In particular, guidance recently generated by the FDA

on highly multiplexed PCR assays provides information on the types of bridging studies or targeted validation studies that can be undertaken when making an alteration to an existing multiplex assay. In this way, all studies undertaken during the original multiplex assay validation do not need to be fully repeated, striking a balance between the need for updated performance data and practical time and cost constraints.

Additional considerations will arise with multiplex assays compared to singleplex assay. These include the complications posed by the potential for probe cross-hybridization, which decreases specificity, and practical challenges with validating mixtures of targets versus a single target. The ability for developers to successfully multiplex assays continues to improve, and there are opportunities for the BioWatch program to move in the direction of multiplexed assays. A critical issue with the prior deployment of multiplex assays in the program was a lack of user confidence in assay performance, compounded by a lack of available performance data. Expanded data sharing (previous recommendation) should help to address this concern and enable stakeholders to reach agreement on whether particular multiplex agent detection assays are acceptable for use in BioWatch.

The committee concluded that existing guidance on singleplex PCR performance for environmental biodetection assays in concert with FDA information should provide a good starting point for the performance testing and validation of multiplex assays by the BioWatch program. The additional issues that arise when designing, validating, and deploying multiplex assays will need to be considered when making the decision to switch to a multiplex format.

INCORPORATING ADVANCES IN SEQUENCING TECHNOLOGY INTO THE PROGRAM

Rapid developments in technology such as NGS have significant implications for the program. The program eventually may be able to move away from the limitations of PCR assays that target only a few small regions of a microbe to a more definitive form of agent identification that relies on information recovered from an expanded portion of the genome. Metagenomic NGS, in particular, is attractive in the future as a universal, target-independent approach to identify novel infectious agents, including emerging pathogens associated with outbreaks (Briese et al. 2009; Grard et al. 2012).

In the short term, the program can use sequencing as a tool to help resolve unusual assay results. In this way, sequencing could function as a tertiary confirmation and characterization tool. Sequencing and associ-

ated informatics are not yet at a stage in which they could replace PCR assays as the program’s principal detection methodology, and many challenges remain in detecting a low-prevalence target sequence against a high-complexity environmental background in a reasonable period of time. However, the program should monitor developments in this area as speed increases, cost decreases, and required informatics and databases improve.

Recommendation

The Department of Homeland Security should monitor developments in technology, such as next-generation sequencing, as speed increases, cost decreases, and required informatics and databases improve. A program to evaluate technology and incorporate it into BioWatch would enable the program to improve its ability to address current challenges and enable it to meet those of the future. However, the program should plan to work with at least some early adopter laboratory users in the BioWatch jurisdictions, along with technology experts, to make sure that new technology brought into the program not only incorporates the best technology but also functions smoothly for the stakeholder community.

BioWatch is only one tool in a layered network of public health and safety disease surveillance activities. The challenges that the program faces are complex, and to be successful the results obtained from the system’s collectors must be useable by decision makers in state and local jurisdictions. Currently, and for at least the near future, the program relies on real-time PCR as the core of its agent detection technology. The program has a strong foundation from which it can develop assay performance and validation standards that meet program needs. As the report observes, however, the most striking issues do not appear to be technical challenges. Rather, they are primarily communication and education and training. Building additional, specific mechanisms into the program to share assay and system performance results, discussing what these performance results reveal about the limits of the assay and the translation of assay results into actions, and sharing applicable assay performance information across agencies such as DHS and CDC and with appropriate individuals within the network of jurisdictions appear to the committee to be critical needs. Even while recognizing the necessary security constraints, this goal should be achievable. Addressing this issue would go far in ensuring confidence in the system and its results in order to facilitate responses to a BioWatch Actionable Result.

This page intentionally left blank.