maintaining or modifying elements of the strategic environment to serve the interest” outlined in U.S. policy (O’Donohue, 2019).

• Policy is a set of guiding principles that outline organizational rules for activities that are repetitive in nature. In short, it outlines what should and should not be done by the organization. According to Joint Doctrine Note 2-19, at the national level, policy documents summarize “the positions of governments and others cooperating, competing, or waging war in a complex environment” (O’Donohue, 2019).

While many strategy documents were reviewed and considered by the committee (see Appendix A, Table A-1 for a complete list), the committee chose to focus its efforts on the eight documents listed in Table 3-1. As a note, the 2023 DoD Strategy for Countering Weapons of Mass Destruction was not subject to the Committee’s evaluation methodology.

3.2 METHODOLOGY OF ASSESSMENT

Well-reasoned and intentioned efforts to ensure accountability, critically important in the context of the use of national security, public trust, and good use of tax-payers money, often drive requirements for metrics (Muller, 2018). One of the biggest challenges can be expressed using Goodhart’s law: “When a measure becomes a target, it ceases to be a good measure” (Strathern, 1997). Ensuring that metrics are meaningful (rather than trivial, political, idiosyncratic, or less than meaningful) is hard. The classic case in national security comes from metrics on success put in place by the DoD for the military services during the Vietnam War (Daddis, 2012). Previous high-level efforts to define “broad-based objective criteria when evaluating progress in the nation’s efforts to combat terrorism” have noted challenges:

This committee has aimed to create and employ an objective, repeatable methodology for assessing the effectiveness of strategies that incorporates metrics and also seeks to ensure that they are meaningful and comprehensive, considering the whole rather than focusing on single metrics or limited cases.

To address the adequacy of strategies to identify, prevent/counter, and respond to chemical terrorism, the committee adopted a methodology that employs a systematic approach to evaluating relevant documents. Appendix D provides the complete rubric. Several documents ranging from national level to individual agencies strategies were evaluated in detail. These were made publicly available within the last 10 years since the formation of this consensus study committee. The adequacy of these strategies was examined through three different lenses—identity, prevention/countering, and response. Before reviewing the documents, a list of characteristics that define an adequate strategy

TABLE 3-1 Key Strategic Documents

was created for each category. Then, each evaluation was measured against the same rubric, which allows for a qualitative yet consistent approach to assessing the strategies across the three major groups. In cases where the strategy does not address the topic, the document was not evaluated.

The rubric addresses the following questions:

1. Does a genuine strategy exist, and if so, to what extent is it coherent?
2. Does the strategy sufficiently meet the chemical threat over the required timeframe of interest? If so:
   1. To what extent do the goal(s) collectively encompass the level and type of threat that is likely to emerge in that timeframe?
   2. What policies, plans, and resource allocations are enabling the goals to be achieved?
3. How feasible is the strategy concerning statutes, fiscal, and politics?

The cohesiveness of the strategy was answered by initially scanning for two key variables: 1) stated goals related to either identify, prevent/counter, or response and 2) clear definition(s) of success for when a goal is achieved. The validity of each stated

goal was further assessed by whether the USG has any policies, plans, or resource allocations available to address them. Each identified goal and its corresponding policies, plans, and resource allocations were also verified for explicit documentation in the strategy and consistency between them. The effectiveness of the goals was also assessed on how well they can respond to a chemical threat that is likely to occur within a relevant timeframe and possibly beyond the nature of the threat. Policies, resource allocations, plans, and their ability to enable the goals were also measured against the timeframe. The legal, fiscal, and political feasibility of implementing various aspects of the strategies were also explored. Finally, the overall adequacy of the strategy was assessed using a qualitative ranking system that ranged from exceeding to inadequate, with partially inadequate, partially adequate, and adequate as other choices within the spectrum. While the committee recognized that this methodology is incomplete and the sample size of documents examined is small, the rubric employed in this assessment was useful for providing a centralized and consistent platform to evaluate the three major categories and communicate the respective findings, conclusions, and recommendations. The committee also found this method valuable for extrapolating findings relevant to de facto strategies.

Chapters, 4, 5, 6, and 7 will discuss in detail the general findings from the strategy assessments and will also include any technical, policy, or resource gaps found that, if included could strengthen parts of the strategies. In addition to the results from this method, the chapters will provide other types of evidence from various sources (e.g., briefing presentations from federal agencies, literature, and congressional hearings).

REFERENCES