Review of the Department of Veterans Affairs Presumption Decision Process (2023)
Chapter: 5 Synthesis of the Presumption Decision Process Assessment
5
Synthesis of the Presumption Decision Process Assessment
The Sergeant First Class Heath Robinson Honoring Our Promise to Address Comprehensive Toxics Act (PACT Act) of 2022 (PL 117-168) and the Department of Veterans Affairs (VA) initiatives that led to the revision of the VA presumption decision process were based on a desire to create a process that is scientifically based, fair, consistent, transparent, timely, and veteran centric. In this chapter, the committee brings together common and overarching themes it identified during its assessment of both the governance (Chapter 3) and scientific (Chapter 4) aspects of the process, given the many overlapping issues. It discusses areas where the presumption decision process is successful in meeting its aims and areas with significant and pervasive problems and where improvements can be made. The committee presents its conclusions that have relevant components for both the scientific underpinnings of the process and governance in the pre-decisional presumption decision process document. Based on its assessment, the committee reiterates its five recommendations, offered in Chapters 3 and 4, that it believes would optimize the clarity, flow, logic, presentation, and transparency of the presumption decision process while ensuring that it is scientifically based, fair, consistent, and veteran centric.
OVERARCHING THEMES
The presumption decision process is not specific to any one era or cohort of veterans and must be able to accommodate a broad range of environmental exposures and medical conditions, including mental health conditions. Therefore, the committee was cognizant that VA’s presumption
decision process and its corresponding documentation could not be too prescriptive but must provide enough detail to encompass a range of conditions, environmental exposures, and conflict eras. It must also be feasible, flexible, and have the appropriate elements to fulfill its purpose and balance a need for brevity with sufficient information to explain and implement it.
The pre-decisional presumption decision process document reviewed by the committee combines two distinct concepts: VA governance operations and the approaches for evaluating the scientific evidence of a relationship between an environmental exposure and a medical condition. The committee identified several overarching concerns with the document: the lack of description such that the committee was unable to judge if scientific best practices are used; weak logical flow; and lack of details, criteria, or standards. Each of these concerns and the committee’s recommendations regarding them is discussed in the following sections.
Use of Scientific Best Practices
The committee’s first task was to examine “whether the process is in accordance with current scientific standards for assessing the link between exposure to environmental hazards and the development of health outcomes.” The scientific aspects of the presumptive decision process are presented in sections 3, 5, 6, and 7 of the document. Overall, the lack of details and specificity in the document (see section Details, Criteria, or Standards), and in the subsequent supplemental documentation and VA responses to committee queries, make it difficult for the committee to judge the scientific appropriateness of the process and whether its elements are fair, consistent, transparent, and veteran centric.
Numerous organizations—for example, the National Toxicology Program (NTP) at the National Institute of Environmental Health Sciences, Environmental Protection Agency (EPA), World Health Organization (WHO), and European Food Safety Authority—use methods, such as systematic reviews, to assess the potential associations between exposure to environmental hazards and the development of health conditions and support their decision-making processes (e.g., EFSA, 2010; EPA, 2022; NTP, 2019; WHO, 2021). In particular, NTP’s systematic review process to assess potential environmental hazards may be broadly applicable to the VA presumption decision process. Although an example of NTP’s systematic review process was given in Chapter 4 because it may be broadly applicable, other systematic review processes may be used or adapted. The committee considers the methods described by these four organizations to be scientific best practices for using systematic reviews to address environmental health research questions, and the committee’s discussion of the VA presumption decision process and document is in that context. Scientific best practices for standardized evidence-based
environmental health assessments, including for mental health conditions, have general steps involving problem formulation (assessment planning), protocol development, evidence identification and selection, individual study assessment (including critical appraisal), body of evidence assessment (including synthesis and evaluation of the confidence in or strength of evidence), and reporting. Several types of evidence review may be used—systematic reviews, rapid reviews, or scoping reviews—depending on VA’s needs and the evidence available. Guidance on conducting these assessments and reviews is available from several organizations, such as the NTP Handbook for Conducting a Literature-Based Health Assessment Using OHAT Approach for Systematic Review and Evidence Integration (NTP, 2019).
Recommendation 4-3: The committee recommends that VA use existing frameworks, tools, and approaches designed for environmental health assessments (e.g., NTP; World Health Organization Framework on the use of systematic review in chemical risk assessment) and apply or adapt them in a manner that aligns with scientific best practices.
To be consistent with scientific best practices, a written work product (e.g., a protocol using PECOTS [population, exposure, comparator, outcomes, timing, and setting], assessment plan, scoping review) is developed as the output of the planning and preparatory phase of the scientific process. This product would identify the information necessary to prioritize, select, and determine how to review each condition and typically includes information gathered from a variety of sources, including database searching for existing reviews and large nonrandomized studies. Existing reviews would include nonhuman studies (e.g., studies addressing mechanistic or biological evidence in animal or in vitro models), if applicable. However, the work product would clearly describe how these study types would be incorporated into the overall evaluation of studies.
The committee concludes that the overall value of the presumption decision process could be enhanced by revising the existing evaluation process in the document and incorporating scientific best practices, either by including additional details in the presumption decision process document or by reference to other documentation.
PECOTS
The evaluation methods of scientific best practices typically begin with developing a structured research question using a PECOTS framework that helps inform and guide all steps in the review process—beginning with evidence selection and ending with developing conclusions that are a direct
response to that question. The PECOTS framework is used to guide problem formulation and the development of a protocol that specifies the scope of the assessment, details the research questions, and outlines the methodology for the evidence review. In scientific best practices, the structured research question and methods would be documented in a protocol before the review.
Section 7A of the VA presumption decision process document is titled “PECOTS Framework”; however, VA’s description of how it intends to use PECOTS and the corresponding PECOTS schematic (see section on Diagrams and Figures in the Presumption Decision Process Document later in this chapter) are not consistent with the standard use of this term. For example, VA states that the condition-specific review panels will use a consensus-based evaluation process adapted from PECOTS to define the strength of the evidence base and that PECOTS “defines the objectives of the review or guideline” and “informs the study design or inclusion or exclusion criteria for a review.” The committee notes that PECOTS is not generally recognized as a consensus-based evaluation process, and VA presents it as more of an evidence-to-decision framework and therefore misuses the term. However, VA did use an appropriate PECOTS framework and description, including an explanatory table of PECOTS elements, in its rule for “Presumptive Service Connection for Respiratory Conditions Due to Exposure to Particulate Matter” (Federal Register, 2021).
The committee finds that, given the variety of conditions to be evaluated and the differences in the amount, type, and quality of the evidence to be reviewed for any condition, the scientific evaluation process will need to be flexible to address the research question that is formulated using PECOTS.
The committee concludes that the PECOTS terminology, the PECOTS schematic (including its content, order, and organization), and the accompanying text in the presumption decision process document are not consistent with scientific best practices.
GRADE
Section 7C “Determination of Levels of Evidence” of the VA presumption decision process document indicates that VA uses a “semi-quantitative approach” to evaluate the quality of evidence and determine the levels of evidence based on the GRADE (grading of recommendations assessment, development and evaluation) structure. GRADE is a tool used to grade the quality or certainty of evidence and determine the strength of the recommendations (Guyatt et al., 2011). It was developed and has been in use for decades primarily in the clinical research setting.
The VA presumption decision process document does not explain how GRADE will be used by subject matter experts to assign certainty ratings to the evidence. Section 7C states that GRADE “considers factors that diminish (e.g., bias, imprecision) and enhance (e.g., large effect, dose-response) the overall quality of evidence.” Applying GRADE to environmental and occupational health questions is a somewhat new concept and practice; although possible, it often necessitates modifications to the framework to incorporate the types of studies and data under evaluation for environmental health (Guyatt et al., 2011; NTP, 2019; Rooney et al., 2014; Woodruff and Sutton, 2011). The use of a structured process to assess the quality or certainty of the body of evidence need not be limited to the GRADE method, as the committee recognizes that GRADE may not fully address aspects of evidence that are important to VA for addressing environmental health questions, such as the use of Veterans Benefits Administration (VBA) claims and mechanistic data.
The VA presumption decision process document, which outlines the standardized evaluation process, does not explain if or how individual studies will be critically appraised for validity and reliability, which is needed to be consistent with scientific best practices. That approach for individual studies would typically be described in a protocol and applied during a standardized assessment.
The committee concludes that more information on what and how data elements are to be extracted in a “standardized review process” would increase the clarity of this critical aspect of the evidence review process. Furthermore, the lack of a formal appraisal of the reliability or validity of individual studies is not consistent with scientific best practices.
Other Considerations for Using Scientific Best Practices
Section 7 “Standardized Evaluation of the Science” of the presumption decision process document briefly addresses general concepts, tools, and approaches that are important for evidence-based assessments. PECOTS and GRADE are specifically mentioned, and the committee acknowledges that these could be helpful tools to facilitate such an assessment. However, the committee stresses that PECOTS, GRADE, and other such tools are not algorithmic. Each tool has various domains and dimensions and inherent characteristics that should be considered and weighted with regard to its appropriateness for use in conducting an evidence-based evaluation.
However, for the reasons described in Chapter 4, the committee concludes that how these concepts are described and the lack of information on how they will be operationalized is not consistent with scientific definitions or best practices.
The presumption decision process document oscillates between providing high-level features of the process (for example, that a notification will be placed in the Federal Register) to very specific information (such as the description of rare conditions in section 6E). Some of its information is not entirely correct (e.g., description of PECOTS) and lacks an appropriate level of detail. Scientific best practices do not require rigid rules or absolute criteria for assessing the evidence of a condition. Indeed, given the variation in the amount, types, and quality of the available information, guidance that addresses data selection, quality, bias, validity, and reliability and how they might be analyzed and synthesized to reach a presumption recommendation may be more practical and would permit the process to be more flexible, adaptable, and scientifically justified. Such guidance has several purposes:
- To set out recognized best practices (including means for avoiding misinterpretations of evidence or introduction of biases),
- To promote consistency among analyses and actions taken under the guidelines, and
- To explain the decision-making process to stakeholders as a matter of transparency.
The committee concludes that although the components and methods important to conducting assessments in accordance with scientific best practices noted in sections 3, 5, 6, and 7 of the presumption decision process document are necessary, viable, and worthwhile, their descriptions are inadequate and need to be articulated in the document itself or provided in other supporting documentation. Furthermore, the components and methods need to be logically organized and described in a manner that is consistent with current scientific definitions and practices.
Recommendation 4-2: The committee recommends that VA model its scientific evaluation of the environmental health evidence using existing standardized and structured approaches. Such a standardized evaluation process should include a formal problem assessment and study planning phase; development of a protocol that addresses the structured research question (e.g., PECOTS) and includes a detailed literature search strategy and inclusion/exclusion criteria; and a report that presents the systematic identification and selection of evidence, critical appraisal of the validity and reliability of studies, synthesis and integration of a body of evidence, and a structured approach to determining conclusions (levels of evidence) about the scientific evidence.
In addition to the need for more explicit criteria and details related to operationalizing the entire presumption decision process, including evidence
evaluation and decision making, flexibility is also important, especially when the amount of information or scientific evidence will differ with each condition. Criteria for how the condition-specific review panels review and assess the evidence and councils and boards review the presumption reports need to be flexible.
On the whole, the committee finds that many of the elements necessary for the presumption decision process to be scientifically based, fair, consistent, transparent, timely, and veteran centric are identified in the document. However, the elements are not presented in the order in which they would be conducted, and detail is insufficient on how they would be operationalized for the committee to determine whether they are in accordance with scientific best practices.
Logical Flow
The presumption decision process document lacks internal inconsistency. The governance and scientific pieces of the process are labeled as distinct sections but are intertwined throughout the document and presented inconsistently and somewhat illogically, which may lead to misperceptions and misunderstandings. The governance process (illustrated in the Governance Schematic in the document and interpreted by the committee in Figure 3-1) can be divided into two activities that represent a progression from data analytics (monitoring and surveillance of internal and external data sources in the planning and preparatory phase and again in the scientific evaluation performed by the condition-specific review panel) to policy considerations (councils and boards to the VA Secretary). Sections 3b, 3c, 4, and 8 of the presumption decision process document that cover aspects of governance are not grouped together. Other sentences related to governance functions or decision making are embedded in the scientific sections. For example, section 7A includes a sentence regarding a benchmark of an 80% majority agreement on a presumption recommendation to be sent to the VA Secretary, although it is unclear at which governance step this agreement is reached.
The panels and councils involved in the scientific-based reviews leading to a decision on whether to recommend a condition for presumption are the Environmental Exposures Sub-Council, condition-specific review panel (selected by cochairs of the Environmental Exposures Sub-Council), and the Evidence-Based Policy Council. The VA entities that are focused on the nonscience (policy, political, and financial) aspects of the presumption decision process are the Evidence-Based Policy Council (dual role with scientific-based reviews), the VA Executive Board, and the VA Secretary.
The committee finds that, on the whole, the VA governance process used to approve the list of conditions to be considered for presumption and the condition-specific presumption recommendation report is reasonable and logical.
The mixing of governance and scientific processes also has implications for nonconformance with scientific standards or best practices. For example, the process as presented in the presumption decision process document does not follow a logical progression from developing the research question (PECOTS) to determining the likelihood of a positive association (equipoise).
Diagrams and Figures in the Presumption Decision Process Document
Three figures were included in the presumption decision process document. The first was a “PECOTS Schematic” (embedded within section 7), the second was the “Presumption Decision Process Flow” (at the end of section 7), and the third was the “Governance Schematic” (at the end of section 8). The first two figures in particular are confusing and questionable, as they do not accurately represent the PECOTS concept or clarify the steps or decision points of the presumption decision process flow, respectively.
PECOTS Schematic
VA’s use of “PECOTS” and the corresponding figure and text (section 7A) in the presumption decision process document are not consistent with the standard use of this term as described under “Use of Scientific Best Practices” in this chapter. Strictly speaking, the dotted box with an arrow originating at the box “Establish Biological Activity in Humans” might be constructed as the actual PECOTS research question for the presumption process. The rest of the PECOTS Schematic figure (reproduced as Figure 4-3) appears to be more of an evidence-to-decision framework, so the term is misused. The PECOTS Schematic includes elements (as solid and dotted boxes) that the committee interpreted to broadly demonstrate how various sources of scientific literature (no VA health or benefits data are included) might be assimilated to determine a level of evidence and the corresponding likelihood of a positive association. It shows, at a very high level, the evidence-to-decision process using multiple evidence streams (not all of which will be available for each condition being considered) but does not show a PECOTS framework and is therefore misused.
Presumption Decision Process Flow
This figure provides a cursory overview of the intended data sources, activities, and timing for the presumption decision process. The document text contains no call-out to the figure as a
whole or to individual boxes or items within a box. The committee labeled each box in the flow diagram with a letter for ease of discussion in Chapters 3 and 4 (Figures 3-2 and 4-1, respectively).
In general, the flow of the boxes is logical, beginning with a preparatory phase (box A), determination to review condition(s) (box B), and VA’s determination of best method to review (box C) by either VA or an external entity. Box D (Steps for Topic) includes a literature search and screening of studies (normally part of the preparatory and planning phase) and selection of reviewers, a systematic evaluation of the evidence (e.g., data extraction), and analysis and synthesis of data with subsequent report findings (which would include developing a recommendation for presumption status). In response to a committee request for clarification of each step in box D, VA stated that “in step 7 the determinations and recommendations would be presented in a report (with recommendations) for referral to the Administrative Reviews in box E” (VA, 2023). Thus, box E item 1, “Peer Review,” might actually be part of box D and would be separate from the administrative reviews performed in box E.
The committee found no one-to-one correspondence between the document sections that pertain to governance or scientific aspects of the process and the boxes and items shown in the flow diagram; furthermore, some of the information in the diagram appeared to conflict with VA responses to the committee’s information requests.
The committee concludes that the accompanying section order and content narrative in the presumption decision process document do not provide sufficient detail to understand the activities in each box of the flow diagram or the items within a box—specifically, how the items are defined or operationalized.
Details, Criteria, and Standards
The issue of lack of detail is a pervasive problem in the entire presumption decision process document, from the planning and preparatory phase (identifying, prioritizing, and selecting conditions for evaluation) to the approval process for a recommendation of a condition for presumption, and is of concern for both the scientific and governance processes. Although the document appropriately includes the major elements, information sources, and considerations for identification and scientific evaluation of conditions, it provides insufficient details on how the process will be implemented. This lack of detail raises questions regarding quality assurance aspects of the evidence review process and exacerbates concerns about its transparency and fairness.
As described in Chapter 3, following the most recent VA-determined presumption decision regarding asthma, rhinitis, and sinusitis (to include rhinosinusitis) in association with presumed exposures to fine particulate matter while serving in the applicable locations of Southwest Asia in support of the 1990–1991 Gulf War and the post-9/11 operations, VA published a 24-page explanation of the decision in the Federal Register (2021) that included the sources of evidence used and what each contributed, such as VA claims data; EPA’s exposure assessment for particulate matter; and VA’s internally conducted literature search. The description also included the PECOTS parameters used, types of studies considered (human and nonhuman), and use of the GRADE framework with accompanying scoring matrix (Federal Register, 2021). Providing such a detailed rationale for each presumption decision, whether that decision is positive or negative, is important and should be continued.
The committee finds that the presumption decision process is not inherently flawed; rather, the incomplete and opaque documentation of it makes it difficult to ascertain whether the process is fair, consistent, timely, and veteran centric.
The committee concludes that the presumption decision process document provides insufficient detail on the governance process for both the review and approval of the list of conditions to be considered for presumption and the review of a condition-specific presumption report. The committee also concludes that there is a lack of detail, specificity, and criteria for how conditions are identified, selected, and prioritized for addition to the list of medical conditions for consideration for presumptive status and for how condition-specific evidence is to be identified, extracted, analyzed, synthesized, and integrated to determine the strength of evidence. Although not all decision-making attributes can be made public, the committee further concludes that a high-level distillation of the process, entities, and criteria for both positive or negative decisions could be made publicly available.
For example, section 3 of the document states that VA will use a number of factors, such as number of veterans affected, severity of the conditions, amount of literature available, and Veterans Health Administration (VHA) and/or VBA data trends, to generate a preliminary list of conditions to be evaluated for presumption status. The committee notes no specific mention of mental health conditions in the sections on condition selection or the components that are part of the review process, which might lead stakeholders to conclude that only physical injuries and illnesses will be considered. When or how these data sources are screened or what criteria
will be applied to them (e.g., how many veterans need to be affected, or how severe a condition must be to be considered) are not presented. Much of this analytical work would presumably be undertaken during the planning and preparatory phase for developing the list of conditions; however, although some data sources are shown in boxes A, B, C, and parts of D in the flow diagram, their applicability to development of the list of conditions versus the preparation of a condition-specific report is unclear. Furthermore, this diagram is not shown until the end of section 7 and not called out in the document text at all. Section 6 gives minimal information about the use of VHA and VBA data, with no details about how these data will be analyzed or whether they will be weighted differently than other data sources, such as peer-reviewed epidemiologic studies.
Given that VHA and VBA data are mentioned in three distinct sections of the presumption decision process document, the committee finds it concerning that there is only a vague description and an almost complete lack of detail on how these data will inform a recommendation for a presumption.
In addition, it was difficult for the committee to meaningfully assess the soundness and acceptability of the scientific aspects of the presumption decision process given the lack of detail in each scientific section of the document. Five sources of evidence are listed in section 6 (health care data, scientific literature, VBA claims data, other factors, and rare conditions). However, the lack of detail on how these very different types of evidence will be identified, selected, and extracted; the criteria that will be used to review them; and how they will inform a presumption recommendation is a persistent shortcoming of the document. It is also unclear whether these data sources will be used to develop the list of conditions and, if so, whether they will be used in the same way with the same review criteria.
The committee finds that leveraging a variety of data and evidence sources to inform the selection of topics is appropriate. However, specifying and making public how each data source (for example, peer-reviewed epidemiologic studies versus VBA claims data) is intended to be used and weighted, if appropriate, and what criteria are used to review them or use them to select a condition for further consideration will enhance transparency. The committee further finds that the presumption decision process document does not contain sufficient information to assess whether the approach or process for extraction of key data elements from evidence sources is consistent with scientific best practices (see “Use of Scientific Best Practices” section).
The committee finds that VA does not define the decision-making criteria for using the evidence synthesis to determine the strength of association between an exposure and a medical condition, particularly given the potential use of disparate evidence streams. The evidence-to-decision process also is not defined.
With regard to the governance process, the committee received some information about membership for each council and board, but no details were provided on the evaluation and decision criteria used by each of them. The lack of information on how the governance steps are implemented and the decision-making criteria or standards at each step does not ensure a process that is high quality, scientifically sound, fair, and consistent. The committee notes that only identifying each panel, council, board, or other governance entity in the review and approval process without specifying the responsibilities of and decision-making criteria for each one (other than to approve or disapprove a list or recommendation) does not sufficiently explain the process or promote stakeholder understanding, particularly by those veterans and their families who may be affected by a presumption decision.
The committee concludes that the presumption decision process document does not provide an overall framework for the process or sufficient detail for the steps within the framework to determine if the process can achieve its intended purpose, as the document does not clearly describe the use of scientific best practices or the criteria for many of the steps in the process.
The committee concludes that a final presumption decision process document that separates scientific and governance aspects to the extent possible, presents each step in sequential order, and provides a sufficient level of detail (for example, membership and criteria or standards used for decision making by each governance council or board, or how data sources will be used and weighted for scientific evaluation) would help to clarify decision factors and decision points along the process, leading to a more understandable, consistent, and cohesive process.
Recommendation 3-1: The committee recommends that VA make explicit the operational criteria or guiding principles for each of the governance steps and provide a description of the expertise and the entities represented at each step. To the extent possible, these criteria or principles and descriptions should be made publicly available either in the presumption decision process document or by reference to other documentation.
Similar to the lack of detail in the governance process, the scientific aspects of the process lack necessary and important details (see “Use of Scientific Best Practices” for examples).
Recommendation 4-1: The committee recommends that VA’s presumption decision process contain sufficient detail to define how it will operationalize each step of the scientific process, either in the presumption decision process document or by reference to other documentation, beginning with condition identification and selection, through evidence review, to the application of a standard on the likelihood of a positive association.
The committee acknowledges that VA desires a compact and not overly technical presumption decision process document; however, the details noted in these recommendations need to be publicly available so that stakeholders can understand, if not agree with, the process for a given condition and the information used to make a decision. Many agency documents incorporate additional details by citing existing documents, but if VA does incorporate these details by reference, it needs to be clear what information is being used from each referenced document.
Equipoise Standards
The threshold for a presumption is defined in the presumption decision process document as the likelihood or probability that a positive association “is at least as likely as not” (termed “equipoise”). The document lists four standards for “equipoise” and their associated definitions: sufficient, equipoise and above, below equipoise, and against. However, the document lacks information on how a “standard” of equipoise is to be operationalized and applied to produce a presumption recommendation for a condition, which is a notable shortcoming. Once criteria to operationalize the application of an equipoise standard are made, VA will need to test them to ensure they are appropriate since no validated method for this application exists. The presumption decision process document does not indicate whether the conclusion and subsequent presumption recommendation made by the condition-specific review panel require consensus or how dissent is addressed.
The committee concludes that the term “equipoise” denotes a lack of consensus across the medical community and that the term as required by law to be used in the presumption decision process is inconsistent with the current scientific use of it.
FUTURE EVALUATION AND ASSESSMENT
After the preliminary list of conditions goes through the governance process for approval by the VA Secretary, it is published as a notice in the Federal Register that explains why the conditions were chosen and solicits input from the public. VA states that public feedback will be used to finalize the list of conditions that will be considered for further evaluation.
The committee finds that using the Federal Register to seek input from veterans and other stakeholders is an appropriate mechanism. However, the presumption decision process document does not specify how VA will use any public comments to inform the final list of conditions to be considered for presumption status.
The committee concludes that making public the process and criteria used for selecting conditions that VA plans to evaluate for presumption status will further the goals of transparency, fairness, and consistency. Improved transparency will also make the process more veteran centric.
The second phase of decision making—scientific review of conditions—begins with formation of a condition-specific review panel and ends with a final decision by the VA Secretary on whether to approve a presumption recommendation. VA states that the presumption decision process itself starts when the evaluation of the scientific evidence is complete and a report with the presumption recommendation for a condition is submitted to the governance process (VA, 2023). Section 4 of the presumption decision process document states that the composition and size of the review panel will be based on the amount of scientific evidence and data available for review. The committee presumes that the amount of scientific evidence on a selected condition is based on the problem formulation (using information collected from monitoring, VA-led literature searches, and any results of externally conducted reviews) that occurs during the condition identification, prioritization, and selection process.
The lack of information extends to the separately convened, condition-specific review panels, as there is no indication of how the expertise for the review panel members (or the peer reviewers) is determined, including the need for differing viewpoints. The addition of a mechanism to seek the perspective of the affected community during the deliberations of the VA review panel will increase transparency and likely increase veteran engagement in and understanding of both the process and its outcomes. Furthermore, it is not clear from the presumption decision process document or any additional information provided by VA who
(i.e., the Environmental Exposures Sub-Council co-chairs, VA Health Outcomes Military Exposures, or another VA office) is responsible for deciding how to review a selected medical condition—that is, whether the review is to be VA led or externally conducted.
The committee concludes that the composition of any review panel needs to ensure that there is sufficient subject matter expertise including core methodologic expertise in the systematic review of environmental health, and that there is a mechanism to obtain the perspective of the affected community. Additionally, the panel would have sufficient capacity to review the totality of the evidence in a timely manner.
The committee concludes that the overall usefulness of the presumption decision process document could be improved by enhancing the existing description of the evaluation process, including detail on how each step will be operationalized, providing a logical one-to-one correspondence between the text and the flow diagram, and incorporating scientific best practices, by either including additional details in the VA presumption decision process document or referencing other documentation.
To ensure that the presumption decision process serves its intended purpose, decisions made using it must be scientifically sound, with clear acknowledgment of the many challenges, assumptions, and uncertainties that go into them. To foster transparency and promote understanding by stakeholders (particularly veterans and their families who are directly affected by those decisions), the public availability of the rationales and justifications for the decisions and the criteria and methods used to make them are vital. As science evolves and methods for evidence-based decision making improve, periodic evaluation of scientific best practices is critical for maintaining an up-to-date presumption decision process. The pre-decisional presumption decision process document does not indicate whether or when any such updates might occur, but it is to be expected that after it has been applied to a variety of conditions, adjustments to the governance process and/or the scientific aspects of the evidence review process, including modifying the data sources or the review criteria, might be necessary. Pilot testing any revised or new methodology to be used in the presumption decision process is an important component of the process as is ensuring that it is flexible and feasible. Therefore, the presumption decision process might not be considered “new” as indicated in the purpose section of the document but rather a type of living document that is able to evolve as does the science.
Recommendation 3-2: The committee recommends that once the presumption decision process has been used by several condition-specific review panels, it be reviewed periodically (by an entity internal or external to VA with the appropriate expertise) to assess whether scientifically based, fair, consistent, transparent, timely, and veteran-centric decisions have been made and whether any modifications to improve the process are necessary.
FINAL OBSERVATIONS
Those responsible for drafting the VA presumption decision process document assembled many well-recognized and commonly used tools that support evidence synthesis and evidence-based decision making throughout the environmental health field. However, these tools were not presented with the completeness, detail, and clarity that the committee believes are necessary to understand and perform the assessments and make well-informed decisions. The committee believes that by reviewing and implementing its recommendations, VA will meet its objective of creating a presumption decision process that supports scientifically based, fair, consistent, transparent, timely, and veteran-centric decisions and that puts the needs of our nation’s veterans first.
REFERENCES
EFSA (European Food Safety Authority). 2010. Application of systematic review methodology to food and feed safety assessments to support decision making: EFSA guidance for those carrying out systematic reviews. EFSA Journal 8(6):1637. https://doi.org/10.2903/j.efsa.2010.1637.
EPA (Environmental Protection Agency). 2022. Draft protocol for systematic review in TSCA evaluations. https://www.epa.gov/assessing-and-managing-chemicals-under-tsca/draft-protocol-systematic-review-tsca-risk-evaluations (accessed May 31, 2023).
Federal Register. 2021. Rule: Department of Veterans Affairs—presumptive service connection for respiratory conditions due to exposure to particulate matter. Federal Register 86(148):42724–42733.
Guyatt, G. H., A. D. Oxman, H. J. Schunemann, P. Tugwell, and A. Knottnerus. 2011. GRADE guidelines: A new series of articles in the Journal of Clinical Epidemiology. Journal of Clinical Epidemiology 64:380–382.
NTP (National Toxicology Program). 2019. Handbook for conducting a literature-based health assessment using OHAT approach for systematic review and evidence integration. https://ntp.niehs.nih.gov/sites/default/files/ntp/ohat/pubs/handbookmarch2019_508.pdf (accessed April 12, 2023).
Rooney, A. A., A. L. Boyles, M. S. Wolfe, J. R. Bucher, and K. A. Thayer. 2014. Systematic review and evidence integration for literature-based environmental health science assessments. Environmental Health Perspectives 122(7):711–718.
VA (Department of Veterans Affairs). 2023. Response to the Committee to Review the Department of Veterans Affairs Presumption Decision Process information and data request. Provided by Dr. Patricia Hastings, Chief Consultant, Health Outcomes Military Exposures, VA. March 10, 2023. Available from the project public access file at https://www8.nationalacademies.org/pa/managerequest.aspx?key=HMD-BPH-22-10.
WHO (World Health Organization). 2021. Framework for the use of systematic review in chemical risk assessment. Geneva: World Health Organization. https://www.who.int/publications/i/item/9789240034488.
Woodruff, T. J., and P. Sutton. 2011. An evidence-based medicine methodology to bridge the gap between clinical and environmental health sciences. Health Affairs 30(5):931–937.
This page intentionally left blank.
