4

The Framework that Surrounds the PCR Reaction Affects Performance

The system that surrounds the polymerase chain reaction (PCR), including the input steps leading to the PCR, the processes for addressing the results obtained and their interpretation, and a comprehensive framework of quality assurance (QA) are critical to overall system integration and to the confidence users have in a PCR assay in an operational context.

SAMPLE COLLECTION, EXTRACTION, AND PURIFICATION AFFECT PCR ASSAY PERFORMANCE

The results obtained through PCR amplification and detection form the basis for subsequent decision making by the users who participate in BioWatch. Characterizing the assays to ensure that they meet performance standards is important to the success of the program. However, even the most perfect assay will be limited by the material that serves as its input. Aerosol samples contain a mix of viable and nonviable microorganisms along with numerous other substances (dust, pollen, dander, etc.). Sample collection, extraction, and purification steps precede the PCR and will affect the sensitivity and specificity of the results that can be obtained (Figure 4-1). This section discusses some of the issues that these steps present for PCR analysis.

Aerosol samples generally have a lower biomass for a given volume than other types of environmental samples, such as water or soil samples. The types of sample collection system used by a biomonitoring

FIGURE 4-1 PCR Assay. Several key steps precede the PCR amplification reaction. When combined with a detection mode, these steps form a complete PCR assay.

program affect the air volumes that can be sampled within a defined period of time and the types of downstream analyses that can be conducted. In the current BioWatch system, air flows across a dry filter, and particulate material above a certain size threshold will be deposited. This system results in desiccation of organisms that become embedded in the filter. Although biological agents that form spores are environmentally hardy and may remain viable, this type of airflow system is less likely to allow recovered microorganisms to be cultured to determine whether they were live at the time of their collection. On the other hand, BioWatch aerosol collector technology uses a relatively large flow rate, enhancing the ability to sample a large volume of air. Other types of potential sample collection technology, such as impaction onto agar plates or into liquid collectors, also exist and would have their own associated advantages and disadvantages for the program. Once a BioWatch filter has been collected, the sample preparation steps (extraction and purification) critically affect the subsequent PCR.

Introduction of Sample Biases

Common extraction methods include soaking samples to release embedded material, often accompanied by mechanical agitation or homogenization (e.g., bead beating) to break up components such as cell walls, and freeze/thaw cycles and detergents to lyse cell membranes and release nucleic acid. Different types of microorganisms will be extracted more or less easily from filters, and may be more or less easily damaged during the extraction process. Spores are hardy, for example, but may require aggressive techniques to break them open and release sufficient amounts of an agent’s DNA. Gram-negative bacteria are more easily lysed, but their genomic material also may be more easily sheared and degraded during extraction. As a result, the specific extraction methods used have the potential to bias the types of organisms that will be most readily available to the PCR amplification and thus be most efficiently detected.

Ability to Remove Inhibitors

A wide variety of substances may be co-extracted with sample nucleic acids. These substances may act in a number of ways to inhibit the effectiveness of a PCR assay. For example, an inhibitor may interfere with cell lysis and reduce the release of target nucleic acid. Once the nucleic acid is released, an inhibitor might degrade DNA so that it is not useable for PCR or might form a complex with it so that the primers, probes, and polymerase cannot access it. Alternatively, an inhibitor might block or affect the functioning of the polymerase enzyme during the PCR, reducing or preventing DNA amplification. For fluorescence-based real-time PCR, contaminants in the reaction also can interfere with successful fluorescence detection.

Substances that have been reported to function as PCR inhibitors include heavy metals and a number of organic or inorganic chemicals. The complex environmental backgrounds expected to be present in BioWatch filter samples increase the potential that inhibitors may be present. For example, the presence of tannic acid, which can be found in some types of leaf litter, and humic acid, which can be found in soils, has been reported to inhibit PCR reactions (Opel et al. 2010; Schrader et al. 2012). The presence of inhibitory substances in environmental samples has been reported to decrease assay sensitivity by an order of magnitude (Wilson 1997). On the other hand, Peccia and Hernandez (2006, p. 3951) report that “[r]eview of PCR application in aerosols suggests that, detection levels of 10–10² gene copies can be achieved provided that appropriate steps to mitigate inhibition have been taken and the cell lysis and DNA purification steps in sample processing are efficient.” Taking into account extraction inefficiencies, detection limits of 2,000-3,000 Gram-positive bacterial cells and 10-25 fungal spores per aerosol collection filter have been reported in the literature (Hospodsky et al. 2010).

As a result, the performance of PCR assays suitable for use in BioWatch needs to be tested against panels of environmental substances and in the context of likely environmental background (such as previously tested filters). The potential impact of inhibitors also informs some of the necessary controls during the PCR, such as inclusion of a control to demonstrate that the reaction mix can successful amplify a ubiquitous DNA target such as the 16S ribosomal RNA gene of bacteria. Various purification steps can be used to separate extracted nucleic acid from other substances prior to use in PCR. For example, extracted DNA can be treated with phenol and chloroform to separate it from co-extracted proteins. However, there will be sample losses and inefficiencies during all of the sample preparation steps.

The importance of sample extraction and purification steps to the sensitivity and effectiveness of downstream PCR raises two issues for consideration by BioWatch:

• Protocols for sample preparation must clearly define the steps to be conducted, reagents and methods to be used, and associated parameters such as times and temperatures.
• BioWatch collectors are deployed in different locations in different jurisdictions across the country, and some are located in indoor settings while others are sited outdoors. It may not be possible to define a single procedure for sample extraction and purification that works for all of these collectors under all circumstances. Rather, a suite of sample preparation protocols may need to be defined along with an accompanying decision framework to inform the determination of which procedures to use where and when.

The BioWatch program provides sample processing and extraction protocols to the jurisdictional laboratories to be used on collected filters. Two extraction options are provided—a protocol using a single tube filter apparatus and a protocol using a 96-well filter apparatus for multiple samples—along with a recommendation for which to use based on the number of samples the laboratory processes. An additional protocol has more recently been provided for high pollen counts; its use is at the discretion of the jurisdictions. The program is considering opportunities to improve these protocols, and it has indicated that such efforts are separate from the PCR assay test and validation that are the subject of the present study. The critical nature of sample preparation steps for PCR assay results is why guidance documents such as those from Food and Drug Administration (FDA) or the Minimum Information for Publication of Quantitative Real-Time PCR Experiments (MIQE) guidelines include sample preparation as part of the assay submission information. This report does not delve in further detail into these PCR input steps, because they were not the focus of the statement of task. However, their importance should be strongly emphasized and be part of any validation of a PCR assay.

Optimization of Reaction Conditions

The reaction conditions during PCR affect performance and may need to be adjusted during assay development. The enzymes used in PCR will polymerize available nucleic acids either perfectly or imperfectly from whatever material is present in the reaction mixture. It may seem counter-

intuitive, but a PCR can have too much DNA or other nucleic acid in the sample. As a result, copies of nucleic acids from an imperfectly matching template can be synthesized by the polymerase, and could be detected as a false positive in some detection formats. This is particularly true when there is an excess of template in the reaction. In other circumstances, too much DNA in the reaction can result in little or no amplification. Reagent balancing is key to PCR and any synthetic reaction. The unknown environmental background in input samples contributes to this issue because background content cannot be controlled. Nucleic acid will be extracted from multiple unknown organisms that make up the material collected on a BioWatch filter. Measures of pre-PCR DNA concentration in a sample should be taken prior to use in a PCR reaction. Simple tests such as use of fluorescent intercalating dyes can be used as a crude DNA sample content quantification method.

Detection Technology

The technology used to detect the amplified PCR products can also affect assay performance. Such detection parameters include the method of detection (e.g., SYBR green DNA binding dye, Taqman fluorescent probes, and hybridization to beads) and the instrument platform (e.g., Applied Biosystems, Roche, or others). In the case of hybridization of PCR products to beads, which is one strategy used in multiplex PCR assays, the PCR products and DNA conjugated to the beads must have high complementarity, resulting in some potential restrictions on the primers and target sequences used. This is an example of a situation in which the detection method can affect PCR requirements. To understand PCR assay performance, the performance must be evaluated on the detection platform that will be used. When an assay has previously been validated on a particular platform and a change is contemplated, bridging studies must be conducted to re-test the assay on the new platform.

VALIDATION IS THE FRAMEWORK THAT SUPPORTS QUALITY

Just as steps that provide input to the PCR are critically important to the overall PCR assay performance, so too are the interpretations that follow a PCR result. As shown in Figure 4-2, a result can lead to the determination that a pathogen’s nucleic acid is present on a BioWatch filter, resulting in further downstream actions, or that a pathogen’s nucleic acid is not present, in which case routine operations continue. Surrounding the entire system from sample collection through post-result decision making, however, is a QA and validation framework. This framework supports robust feedback to the PCR assay (and other steps in the process) in order

FIGURE 4-2 PCR assay performance is influenced by feedback from the steps that follow a result and the surrounding QA infrastructure.

to further inform the understanding of assay performance, detect issues that arise, determine the root cause of such issues, and implement appropriate corrective action plans.

The Process of Validation: General Principles

Validation is the process of assuring and documenting that a thing, such as a test, device, or process, fulfills the purpose for which it is intended. As defined by Budowle et al. (2014, p. 4), validation accomplishes the following goals:

1. Assesses the ability of procedures to obtain reliable results under defined conditions;
2. Rigorously defines the conditions that are required to obtain the results;
3. Determines the limitations of the procedures;
4. Identifies aspects of the analysis that must be monitored and controlled; and
5. Forms the basis for the development of interpretation guidelines to convey the significance of the findings.

A full validation pipeline includes multiple steps, from defining intended purpose through post-implementation assessment. Validation is not performed one time, prior to implementation, but is a continuous pro-

cess to ensure that the test is performing as expected (i.e., part of the continuous quality control/quality assurance [QC/QA] process). Using validated approaches is a critical part of providing confidence in the results to the program’s users. In the context of BioWatch, validation provides an understanding of the strengths and limitations of the data obtained from the PCR assays, which is crucial in interpreting results and using them as a basis for taking action. It is important to note that validation is not the same as setting the performance standard for an assay. The performance standard sets the criteria that the assay must fulfill in order to be able to meet its intended purpose. Validation is the process to ensure that a given assay meets or exceeds those requirements. Extensive guidance exists on the basic validation process and criteria for validating diagnostic assays designed to detect microbial agents for human disease (CLSI 2004, 2012; Jennings et al. 2009; Burd 2010), veterinary infectious diseases (OIE 2013), pathogen detection in food (FDA 2011), and law enforcement investigations using microbial forensics (Budowle et al. 2008, 2014).

Validation includes, but is not limited to, characterization of the analytical performance of the assay when used according to a given preparative method and instrument. As discussed in Chapter 3, measures to characterize the performance of PCR assays include, but are not limited to, limit of detection, sensitivity, specificity, repeatability, and robustness and are usually carried out by one or more independent laboratories to complement the characterization work done by the assay developer. In addition to analytical validation, validation also undertakes characterization of the performance of the assay in an operational context. For veterinary or clinical diagnostics use of a real-time PCR assay, factors such as the assay’s diagnostic sensitivity, diagnostic specificity, and positive and negative predictive value will be examined (OIE 2013).

The series of steps in the validation framework, for real-time PCR detection assays applicable for BioWatch, are captured in Figure 4-3. As can be seen in the figure, after studies conducted by the developer and independent laboratories to validate assay performance characteristics, verify that user labs can obtain appropriate assay performance, and validate and verify the assay in a system operational context, validation includes ongoing monitoring of assay and operational performance. This monitoring includes laboratory proficiency testing, which helps to ensure that samples are handled and assays performed correctly according to procedures. In addition to obtaining sample results, this testing may deal with aspects such as sample entry, report writing and interpretation, and technical and administrative review. But a robust QA program does more than this. The data obtained feed back to inform the understanding of assay performance over time. Assessing the performance of positive controls over time can be an invaluable way to monitor the assay to pro-

FIGURE 4-3 Steps in assay and system validation.

vide such feedback. The results may influence changes to assay design and to the strategies used for testing and validation. The data also serve to inform an understanding of assay and system performance limits and guidance around the interpretation of results for decision making.

CONTINUOUS MONITORING AND ASSESSMENT IS A NECESSARY PART OF VALIDATION

The Role of Quality Assurance

QA is a comprehensive infrastructure that defines the overall laboratory system including validation, maintenance of standard operating protocols, training and testing standards, proficiency testing using quality-controlled reference materials, audits, and corrective action plans.¹

___________________

¹ Quality assurance and quality control are related but distinct concepts. Quality assurance is a comprehensive system and infrastructure that ensures that the laboratory meets standards of quality. Quality control, on the other hand, focuses on the performance of specific assay testing activities.

In 2011, the Department of Homeland Security (DHS) implemented a QA program for the BioWatch program. The program is based on ISO 17025 standards and includes an analytical component and a field component, with yearly reviews. The QA program includes (1) plans and audits, which define requirements and monitor compliance with those requirements; and (2) performance assessment, which evaluates whether performance is at a level to meet program objectives.

The performance assessment incorporates data from the BioWatch jurisdictional laboratories arising from daily, external QC samples that are sent to the laboratories as well as the internal QC samples that are routinely run with the assays. Each laboratory is required to track its data using control charts. An external contractor, Signature Science LLC, evaluates the data acquired from the laboratories to determine if the laboratory is meeting the established metrics and goals; data are examined by Signature Science and DHS based on 8-week sliding windows and each laboratory is reportedly audited every 2 years with the findings provided to the laboratory and to DHS. Information associated with the BioWatch program, including data obtained through the QA program, is available to laboratory directors through a BioWatch portal. In general, a BioWatch laboratory has access to its own data and to trend data compiled across the network, but not to specific data obtained from other laboratories. Proficiency testing is also conducted 3 times per year. As with the BioWatch screening assays, the Department of Defense (DOD) Critical Reagents Program (CRP) is the source for the reagents, QC samples, and proficiency test samples. Benchmarks have been established for false positive and false negative rates (starting from the filter extraction step), which, when linked to confidence intervals for those error rates, determine whether performance goals are being met. This evaluation strategy provides performance reviews of gene targets across the various testing laboratories and of various laboratories across gene targets (Isbell, 2014; Walter 2014b).

Assay Ruggedness

Data from the QA program provide information that can inform user understanding of BioWatch assay performance. For example, the robustness or ruggedness of an assay refers to the ability of the method to achieve acceptable performance under the variable conditions that come with routine use (e.g., the use of different batches of reagents, performance of the assay by different operators in different laboratories, or small instrument-to-instrument differences even when users are running the assay on the same instrument platform). Data obtained through the QA program enable monitoring for changes or variations in the false posi-

tive and false negative rates obtained during QA and proficiency testing, which can provide a multiyear assessment of how robust the assay is.

The continuous performance monitoring stage is also the time during which real-world assay performance issues are likely to arise that may have significant effects on the confidence that jurisdictional decision makers have in BioWatch results. The system can help identify developing issues in the performance of an assay that may need to be addressed through changes to the assay signatures, primers, probes, strain panels, or other features (requiring new performance testing) or through changes to the decision-making algorithm.

Signature Maintenance

One of the practical implications of the increasing genetic knowledge and recognition of microbial genetic diversity, discussed in Chapter 3, is the need for maintenance of the signatures used in the assay and the existence of signature erosion. As the amount of genetic information continues to increase from the identification and sequencing of new strains of a species and of near-neighbor organisms, signatures that were once thought to be uniquely identifying for a particular strain may no longer be, in fact, unique. Such results lead to erosion of the utility of the signatures used in PCR assays, but also provide an opportunity for refinement and improvement of the system to be more selective for the specific strains that pose a threat to public health. As a result, a real-time PCR assay for pathogen detection is not a static, one-time process of development, validation, and deployment.

At regular intervals, the signatures used in such assays must be screened in silico against databases of genomic sequences to flag potential issues such as cross-reactivity with newly identified near neighbors or for the inability to detect a new pathogenic strain added to inclusivity panels. This testing can identify potential false positive and false negative situations, with follow-up laboratory testing carried out if needed. Lack of proper PCR signature maintenance can lead to overconfidence in signatures because “knowable” false positive and/or false negative events have remained unknown. The need for periodic in silico and laboratory re-testing of expected assay performance should be part of an assay performance standard suitable for BioWatch. In silico testing should be undertaken at least annually.

Other implications for assay performance standards are

• Each tested BioWatch agent is likely to require a different number of strains of the agent and of near neighbors to enable adequate testing. One size definitely does not fit all when it comes to testing these PCR assays.

• Both the sequence of the signatures used in the assay and the composition of appropriate inclusivity and exclusivity strain panels can be affected as new microbial genetic knowledge is obtained.
• Appropriate strain panels for highly diverse organisms such as Burkholderia spp. may be significantly different when the assay is used for screening versus for secondary testing. For example, a Stakeholder Panel on Agent Detection Assays (SPADA) working group previously developed a strain panel with 60 well-characterized strains (AOAC 2011 in preparation-a,b). However, the chair of that panel provided the caveat that a final PCR assay used for confirmation would need to be tested against over 1,000 strains to ensure its ability to correctly distinguish strains (Keim 2014).
• A clear process is needed for conducting the regular assay retesting required and for updating versions of the assay and the performance standard so that signature erosion is clearly documented.

The issue of microbial genetic diversity and how to account for it is relevant for any nucleic acid–based detection technology, including the real-time PCR assays currently used by BioWatch. As more information is obtained about the numbers and types of microorganisms in the environment and as increasing amounts of sequence data for these microbes become available, assay design, performance characterization, and validation become more complex. The implications of microbial diversity and the potential for fewer good signatures and greater numbers of panel strains also provide an incentive for programs such as BioWatch to look at the applicability of advancing technologies such as next-generation sequencing (NGS), discussed in Chapter 5.

APPLYING VALIDATION PRINCIPLES TO ASSAY PERFORMANCE CHARACTERIZATION: PUTTING THESE INTO PRACTICE

Chapters 3 and 4 discuss the validation process and principle steps in establishing PCR assay performance. The following section illustrates how these could be applied to BioWatch to provide a reasonable amount of statistical confidence in the results obtained through laboratory assay characterization and validation, while seeking to keep numbers of replicates and numbers of strains practical. The approach discussed below does not duplicate the full level of detail contained in guidance documents such as the AOAC validation guidelines, SPADA Standard Method

Performance Requirements, Public Health Actionable Assay (PHAA) and Federal Standards for Assay Performance and Equivalency (FSAPE) documents, and other sources, which necessarily contain comprehensive and specific explanations for the ways that tests are carried out and data is handled and reported. These represent valuable resources to consult in fleshing out the details of a BioWatch standard.

The following process represents an approach that can serve as a starting point. The basic dimensions are given below and are explored in an example. It is important to emphasize that the specific details of the process ultimately used by the BioWatch program will need to be determined by the program and established in consultation with its stakeholder community.

The committee also noted that the list of pathogens for which BioWatch tests is based on risk analyses and so far has not included RNA-based agents. Should RNA agents be included in the future, the standards for assay validation would be similar to those for agent detection assays for DNA-based microorganisms. In such a situation, the reverse transcriptase (RT) enzymatic step would be considered part of the PCR reaction and the need to optimize and test reaction conditions for potential inhibition also would apply to the RT enzyme.

Starting point for standards approach recommended by the committee :

• Determination of limit of detection. A reasonable approach to determining an assay’s analytical limit of detection (LOD) in a laboratory setting is to conduct serial dilution at a range of concentrations bracketing the estimated LOD. This reflects guidance on LOD determination provided by the Clinical and Laboratory Standards Institute (CLSI), and is also in line with assay data packages through DOD and the PHAA and FSAPE documents.
• Expanded in silico testing. Use of in silico screening presents a relatively fast and low-cost way to predict assay performance using as many strains as are available as high-quality sequences in reference databases. It is also possible that in silico screening may uncover reference database curation issues. These could include pathogen DNA erroneously incorporated into host genomes (or vice versa), contaminated or mixed pathogen genomes, or misidentified pathogen genomes. The lack of ground truth must always be considered when evaluating DNA signatures for pathogens.
• Strain panels to use for testing. The panels recommended by SPADA represent a good starting point and were developed with stakeholder participation. All strain panels need to be regularly reviewed and updated to account for new genetic knowledge. As

• a result, it should be recognized that the current SPADA panels may not provide sufficient genetic coverage for certain pathogens without additional review and modification.

• Testing of inclusivity and exclusivity strains. The strategy used by Los Alamos National Laboratory (LANL) for testing detectability or nondetectability of inclusivity and exclusivity strains represents a reasonable model for BioWatch.
• Cost. Initial costs for assay development and primary performance characterization should be undertaken by the developer. Independent validation should subsequently be conducted by DHS.

This approach is spelled out in greater detail in the example, below.

Hypothetical Example: The BioWatch program decides to deploy a new set of assays against a pathogen of interest.

Assay Development (for a New Assay): The assay developer selects the regions of the tested pathogen to be targeted for detection and designs associated primers, probes, PCR conditions, and other aspects of the assay protocol.

• The developer uses best practices in PCR assay design and optimization to create one or more assays that will detect the pathogen of interest under appropriate reaction conditions, while not cross-reacting with organisms that are not of interest. As part of this process, the developer undertakes in silico assay performance testing by screening for predicted inclusivity, exclusivity, and environmental background performance against organisms whose high-quality sequences are available in reference databases. This approach presents an opportunity to anticipate performance against as many relevant strains or organisms as possible.
• If the sequences are available, in silico performance predictions should be examined against the full set of strains and organisms recommended in the PHAA panels (under the section describing testing of environmental samples, not clinical samples).

Initial Performance Validation: The developer, or a laboratory contracted by the developer, determines assay parameters such as limit and dynamic range of detection, sensitivity, and specificity, along with associated confidence limits on the results.

• The developer obtains the necessary nucleic acids for the strains that will be tested from a repository that is able to supply quality-controlled materials. For BioWatch, this repository could

• be the CRP. If necessary materials could be provided by sources or repositories other than CRP. DHS will need to ensure that those conducting validation testing have sufficient access.

• A primary reference strain and three to five additional reference inclusivity strains are used in determining several of the performance characteristics. These strains should be specified in the performance standard and selected from among the inclusivity strains that the assay should detect.
• The developer determines the LOD for the primary reference strain. If there is agreement on a specific acceptable minimum detection level to be fit for purpose (AMDL, in the case of a standard such as SPADA), testing to determine probability of detection around that concentration could alternatively be undertaken.
• A reasonable approach to determining an assay’s analytical LOD in a laboratory setting is to conduct serial dilution at a range of concentrations bracketing the estimated LOD, using n = 60 replicates with acceptance criteria of at least 58/60 for a given concentration, followed by appropriate curve fitting. If the assay is able to detect 60/60 samples, there is an approximately 95 percent probability that the assay will detect that quantity of DNA with an associated lower confidence limit of 95 percent. If the assay is able to detect 58/60 samples, there will be an approximately 90 percent long-term probability of detection at that concentration of DNA, with an associated lower confidence limit of 95 percent. This approach represents a fairly stringent characterization of assay performance for this variable—as noted earlier, for example, the long-term probability of detection at a particular quantity of DNA must be 99 percent in order to achieve 60/60 detections with greater than 50 percent probability (DHA 2013a).
• A smaller number of replicates could first be used to narrow the expected range within which the LOD falls, with a larger number of replicates used around the hypothesized LOD (an approach that is used in PHAA and FSAPE guidance).
• The developer tests the LOD of the three to five additional reference inclusivity strains at a subset of values around the calculated reference strain LOD to examine detection variability across strains.
• Targeted laboratory testing is undertaken to validate the predicted inclusivity, exclusivity, and environmental background performance against a prioritized subset of strains based on morbidity or mortality, transmissibility, and host range (inclusivity panel) and on sampling relevant genetic and environmental background diversity (inclusivity, exclusivity, and environmental panels).

• The SPADA strain panels represent a reasonable starting point, updated as needed to reflect current genetic information and prioritization since the panels were developed.

• Characterizing detectability of inclusivity strains or nondetectability of exclusivity strains and environmental organisms by testing n = 20 replicates, followed by testing 20 additional replicates if an unexpected detect or nondetect result is obtained, with acceptance criteria of 20/20 or 39/40, represents a reasonable model. If the assay is able to detect 20/20 samples, the 95 percent lower confidence limit on the probability of detection of the reference strain at this concentration will be approximately 86 percent. If the assay is able to detect 39/40, the 95 percent lower confidence limit on the probability of detection of the reference strain at this concentration will be approximately 89 percent.
• In addition to LOD and specificity tests conducted using purified nucleic acid, tests performed in the presence of realistic background matrix, such as by spiking DNA onto used BioWatch filters, is conducted to investigate whether assay performance is affected. For example, a reduced signal could indicate inhibition due to the presence of substances such as metal ions or humic acids. An understanding of potential inhibition due to background is important is assessing negative results. Initial assay robustness also is conducted by the developer to understand performance in the context of variable conditions that may be encountered in use and that may impact PCR. Appropriate positive and negative controls should be included in all of the testing, as discussed in Chapter 3.
• The developer should report the data obtained and accompanying calculations in an acceptable format so that it can be understood and used by those who will perform the tests, those who will interpret the results, and other stakeholders that may have to respond based on the results.

Independent Validation: DHS undertakes or contracts with an independent laboratory (not the developer) to validate the performance information on the assay. Similar steps are repeated, but the focus is on targeted testing of the documented assay performance.

• The LOD is validated using nucleic acid with an appropriate number of replicates (see above). In this case, testing should be conducted in the context of the expected sample matrix.
• Validation testing is conducted to confirm the assay performance reported by the developer against the strains used in the initial

• laboratory inclusivity, exclusivity, and environmental background testing (see above).

• BioWatch uses certain assays for screening and other assays for secondary verification that a tested pathogen’s nucleic acid is present on a collected filter. Although the same core performance standard and validation framework is appropriate for these PCR assays, “tiers” of validation testing could be implemented that emphasize different parameters based on intended use. Laboratory testing using the full sets of PHAA panels is not required for a screening assay, although current SPADA panels may not capture sufficient genetic diversity for all pathogens, and all panels must be revisited regularly. Laboratory validation for a secondary assay could include testing of additional inclusivity, exclusivity, and environmental strains (e.g., drawn from PHAA/FSAPE panels) in order to sufficiently characterize assay sensitivity for this type of intended use.
• Alternatively, a set of subsequent assays (for use following the screening assay) that draw on PCR, NGS, and other technologies could be established for each tested BioWatch agent and designed to serve different roles in ruling in, ruling out, confirming, or characterizing the sample.

When Pulling an Existing Assay Off the Shelf for Deployment: A relevant assay and accompanying package of performance data may have previously been developed, perhaps for another purpose, but not deployed by BioWatch. The validation data generated for that assay could be used to support the use of the assay. In this case, new genomic information may have been obtained since the assay was designed. An in silico reexamination of expected assay performance is carried out to look at the target region of amplification, primer and probe sequences, and anticipated performance against inclusivity, exclusivity, and environmental strains in order to check for potential performance issues prior to conducting additional validation testing as required using the framework described above.

System Characterization and Validation: Any assay must work in concert with all of the steps in the system from sample collection to data analysis and ongoing quality assurance. Similar to the independent validation stage, above, targeted testing to characterize performance should be conducted in a more realistic operational context (e.g., to determine LOD and performance against inclusivity, exclusivity, and environmental background in an operational system context).

• Testing is conducted using samples in the context of the collection media, matrix, and realistic environmental background.
• Sample extraction and preparation procedures are tested from realistic environmental background to determine performance; changes to sample extraction, preparation, or assay procedures made as necessary.
• Assay performance is tested against a panel of environmental substances that might interfere with PCR amplification and detection. SPADA environmental substances panels represent a reasonable starting point for testing. Utilizing BioWatch filter samples from jurisdictions that have experienced PCR inhibition problems in the vicinity of particular BioWatch sampling units would also be useful.

Verification of Operational Performance by User Laboratories: The performance of the assay (either in isolation or, more appropriately, in the context of the system from sample collection to output) should be tested by the BioWatch jurisdictional laboratories that will use the assay. This testing indicates whether laboratories across the program network can achieve the anticipated assay performance results and gives a sense of the operational robustness of the assay.

• This stage determines whether there are local issues (e.g., PCR inhibition, environmental cross-reaction with some or all assays for the organism due to unknown near neighbors, etc.) that may arise and, if so, they will need to be addressed.
• Verification of LOD could be conducted using n = 20 replicates; verification of detectability or nondetectability of the prioritized inclusivity, exclusivity, and environmental strains could be conducted using n = 5 replicates for each tested strain.

Monitoring and Maintenance: Validating and deploying an assay is not a one-time event.

• Operational performance by user laboratories should be monitored over time through a QA program, which may flag potential issues that develop in particular locations or under particular conditions or from lot-to-lot variation.
• New genetic knowledge of relevant microorganisms and of environmental background is rapidly being acquired. DHS should reexamine each assay at regular intervals, ideally annually. This review should include in silico testing of expected assay performance for the target region of amplification, primer and

CONSTRAINTS OF THE CURRENT SYSTEM AND OPERATIONAL EXPERIENCE OF BIOWATCH JURISDICTIONS

Building on the detailed discussion of analytical characteristics that define the performance of real-time PCR assays and the factors that need to be considered for a BioWatch-suitable standard, it is important to return to the real-world context in which these assays are used and the experiences of BioWatch jurisdictions. It is the laboratorians in the network of state and local public health laboratories who have to be confident that they sufficiently understand assay performance and have confidence in the results and the local and national BioWatch Advisory Committee members who need to use assay results in actionable decisions.

The implementation of the BioWatch QA program is a positive recent development to provide guidance. The BioWatch program also conducts periodic workshops or webinars for the network of jurisdictional laboratories in order to discuss the obtained performance data and appropriate interpretation. The QA program thus is a critically important component of the program management processes. However, the committee observed several areas during its data-gathering where it appears that user jurisdictions would benefit from additional discussions and information. An increased effort in communication, education, and training would be invaluable for interpretation of results and building user confidence. Information from these discussions also needs to be translated back to the public health officials who make decisions based on the laboratory results, in order to inform their understanding of the information they receive.

Additional Parameters Influencing Decision Making on Assay Results

To support actionable decision making based on assay results, it is helpful to examine an assay’s sensitivity, to the extent possible, in the context of the pathogen dose required to cause disease. For example, if an assay is able to detect a single bacterium with sufficient confidence, but the infectious dose of the pathogen would require exposure to at

least 1000 bacteria and the assay result yields a threshold cycle (C_t) value around the cutoff cycle (therefore, barely above the threshold for judging it a positive result), a jurisdiction may respond differently. Likewise, if the same assay has an LOD of 100 bacteria (a less sensitive assay), but is producing a very strong signal, the laboratory may respond rapidly and aggressively.

The interpretation of assay results in this context is significantly complicated by the fact that it is extremely difficult to link an assay’s analytical sensitivity with the aerosol quantity of a pathogen. If a BioWatch collector was sampling the very edge of a plume, it may indicate only a weak signal when in reality the center of the plume could have clinically significant pathogen concentrations. So it is inherently problematic for jurisdictions to disregard weakly positive results if they are over the threshold for a positive detection. This is also an opportunity to raise the question of guidance on whether and how to draw on the semi-quantitative information provided by the real-time PCR curves and resulting C_t values, as opposed to yes/no decision making based on cutoff C_t thresholds. Because the screening assay tests for one signature and the secondary verification or confirmatory assay tests for several additional signatures, situations also can arise in which some signatures are positive and some are negative. The program has decision algorithms (e.g., that all 3/3 signatures must give positive results to declare a BAR), but the committee’s impression is that jurisdictions would like to have further discussions on what these types of “partial” results truly mean. The committee agrees. Discussions among the BioWatch program, federal experts, and jurisdictional technical experts on additional guidance on assay interpretation for situations beyond simply detect/nondetect could help inform operational decision making.

Troubleshooting Assay Issues Encountered by Jurisdictions

The operation of the QA program and interpretation of the resulting data could be even more closely integrated with the network of BioWatch laboratories in order to better troubleshoot key operational issues that are encountered by individual laboratories and to provide guidance to the laboratories on the interpretation of results. For example, the issues encountered in jurisdiction “X” with relatively high rates of positive results in the assays for agent “C” do not involve false positive or false negative results on known QC test samples, but should be addressed as part of the QA framework because the issue involves overall performance and validation. The jurisdiction still needs a basis from which to address its specific situation. The reasons an assay does not work well in a particular jurisdiction could include

• Lack of understanding or knowledge of assay performance parameters, which are needed in order to understand assay limits and data interpretation;
• An environmental background issue in broad or limited regions within certain jurisdictions; and
• A technical issue with the performance of the assay.

The most effective solutions to address the different causes above will vary. Similarly, although a significant amount of useful data is obtained through the BioWatch QA program, state and local users may not always know what specific corrective actions have been implemented in response to issues identified in their own or in others’ laboratories.

The impression that the committee obtained (recognizing that experiences may not be the same for all users and that there may be relevant information to which the committee did not have access) is that laboratories and jurisdictions address issues with assay results on an ad hoc basis. Incorporating a systematic mechanism into the QA system for feedback and for making and communicating decisions on whether to implement assay or operational changes would strengthen the already valuable effort. BioWatch needs to have a process to identify the root cause of an issue, discuss appropriate strategies to resolve it, and carry out processes to address it in an expeditious manner. For example, one suggestion might be modeled on the process of medical peer review, in which clinicians periodically come together to review cases and discuss what went right and what went wrong. The BioWatch program could consider holding sessions with federal and jurisdictional laboratory experts and public health decision makers to report on unusual assay results obtained during operational performance and discuss collectively how to solve them. If there is an environmental background detection issue in a jurisdiction, this might provide an opportunity to make a change to increase the specificity of the assay(s) in question that could eventually spread to other jurisdictions.

Communication Across the Program Network and with Additional Stakeholders

The QA operations and information largely involve the initial screening assay. The external contractor, who analyzes the data and provides the information to DHS for its further analysis and any decision making, does not have access to information from the Centers for Disease Control and Prevention (CDC), which is the source for the secondary assay. Because the initial and secondary assays currently are used as a two-step process

by BioWatch laboratories, further communication and regular data sharing would be helpful.

BioWatch jurisdictional users also expressed the strong desire to have greater access to specific assay performance and validation data on both the initial screen and the secondary assay, and to have more detailed QA program data than are currently provided. The depth of laboratory technical expertise may vary from jurisdiction to jurisdiction, but in fundamental ways the BioWatch program operates locally. There are opportunities to better leverage the expertise that exists in, for example, the state and local public health laboratories. Laboratory directors and senior personnel in jurisdictions must present assay results and their interpretation to local BioWatch Advisory Committees and decision makers; these personnel express frustration in not having access to as much performance and validation data as possible to support their responsibilities (personal communications from jurisdictional scientists and officials, September 3-4, 2014).

This issue is not entirely within DHS’s ability to solve. It will involve discussions with the jurisdictions but also with agencies such as CDC. Both BioWatch and the CDC LRN currently maintain user web portals as information repositories, which might be expanded to share additional data.² The need to maintain appropriate security around assay and operational performance information should be taken into account. Nevertheless, mechanisms must be implemented to convey relevant information while maintaining necessary security measures. Sufficient numbers of personnel in BioWatch laboratories should have (or have the ability to obtain) the necessary security clearance, and systems could be devised to share and discuss details of assay analytical and operational performance and validation. Greater transparency and data sharing would improve user confidence. Overall, the committee concluded that a good framework for the program is in place, but the importance of communication to confidence and actionability should be strongly emphasized.

___________

² For example, the LRN portal includes data on the performance of LRN assays (such as LOD, sensitivity, specificity, and matrix) that resulted from testing conducted around 2008 by CDC using inclusivity and exclusivity strain panels assembled by CDC subject experts (this testing predates the current effort to submit LRN assay data to FDA) and would be available to LRN laboratory directors (Toby Merlin and Harvey Holmes, CDC, personal communication, December 2, 2014). The BioWatch portal similarly contains various data accessible to the directors of the state and local laboratories that perform BioWatch assays.

This page intentionally left blank.