Summary

In recent years, the U.S. federal government has invested approximately $463 billion annually in interventions¹ that affect the overall health and well-being of children and youth, while state and local budgets have devoted almost double that amount. The potential returns on these investments may not only be substantial but also have long-lasting effects for individuals and succeeding generations of their families. Those tasked with making these investments face a number of difficult questions, such as:

• What does it cost to implement this intervention in my particular context and what are its expected returns?
• To what extent can these returns be measured in monetary or nonmonetary terms?
• Who will receive the returns and when?
• Is this investment a justifiable use of scarce resources relative to other investments?

Ideally, decision makers would have available to them the evidence needed to answer these questions, informing their investments and increasing the investment returns. Economic evidence² in particular has great

___________________

¹ The term intervention is used to represent the broad scope of programs, practices, and policies that are relevant to children, youth, and families.

² In this context, economic evidence refers to the information produced from cost and cost-outcome evaluations, including cost analysis, cost-effectiveness analysis, and benefit-cost analysis.

potential to show not just what works but what works within budget constraints. (Box S-1 defines the methods used to produce economic evidence that are the focus of this study.) As the result of a number of challenges, however, such evidence may not be effectively produced or applied. These shortcomings weaken society’s ability to invest wisely and also reduce future demand for this and other types of evidence.

In this context, the Institute of Medicine and the National Research Council, in fall 2014, empaneled the Committee on the Use of Economic Evidence to Inform Investments in Children, Youth, and Families. In this report, the committee highlights the potential for economic evidence to support these investments; describes challenges to its optimal use; and offers recommendations whose implementation can promote lasting improvement in its quality, utility, and use.

IMPROVING THE USE OF ECONOMIC EVIDENCE IN DECISION MAKING

While many decisions about investments in children, youth, and families would be enhanced by stronger evidence, including economic evidence, decision makers face budget constraints, time limitations, and competing incentives that limit their use of such evidence. The committee proposes that to overcome these limitations, both producers and consumers of economic evidence give full consideration to two simple but fundamental guiding principles: (1) quality counts and (2) context matters.

Quality Counts

The committee identified challenges to the quality of economic evidence that limit its utility and use. For example, high-quality evidence can be difficult to derive because economic evaluation methods are complex and entail many assumptions. Moreover, methods are applied inconsistently in different studies, making results difficult to compare and use appropriately for policy and investment decisions. Furthermore, the evaluation results may be communicated in ways that obscure important findings, are unsuitable for nonresearch audiences, or are not deemed reliable or compelling by decision makers.

Based on its review of the landscape of economic evaluation, the committee produced a set of research conclusions. These conclusions determined that conducting an economic evaluation requires careful consideration of a number of assumptions, decisions, and practices to produce economic evidence that is of high quality. For example, high-quality economic evaluations are characterized by a clearly defined intervention and a well-specified counterfactual; a previously established perspective, time horizon, and baseline discount rate; accurate cost estimates of the resources needed to replicate the intervention; and consideration of the uncertainty associated with the evaluation findings. In addition, the committee concluded that registries can increase uniformity of practice, and that the acknowledgment of equity concerns can enhance the quality and usefulness of economic evaluations.

Context Matters

Economic evidence, even of the highest quality, may not be used effectively to inform investment decisions if it is deemed irrelevant, infeasible, or difficult to interpret by its consumers. Yet, evidence is often produced without considering the end-user’s needs, values, and capacity to access and analyze the evidence—that is, the context for evidence use.

From its review of the salient research, the committee drew a set of

conclusions about the utility and use of evidence to inform investments in children, youth, and families. For example, the infrastructure for developing, accessing, analyzing, and disseminating research evidence often has not been developed in public agencies and private organizations; interactive, ongoing, collaborative relationships between decision makers and researchers and trusted knowledge brokers are a promising strategy for improving the use of economic evidence; and that growing interest in performance-based financing is likely to increase the demand for economic evidence to inform decisions on investments in children, youth, and families. Moreover, whether evidence is used varies significantly according to the type of investment decision being made and the decision maker’s incentives (or lack thereof) for its use. In short, the committee determined that economic evidence has the potential to play an influential role in the decision-making process—if the concerns and interests of decision makers are considered in the development and communication of evidence.

A ROADMAP FOR MOVING FORWARD

Many of the challenges to the quality, use, and utility of economic evidence affect its consumers, producers, and intermediaries³ alike. Accordingly, the committee formulated a roadmap for promoting improvements in the use and usefulness of high-quality economic evidence. This roadmap highlights the need to foster multi-stakeholder partnerships and build coordinated infrastructure to support the development and use of economic evidence.

The committee concluded that long-term, multi-stakeholder collaborations that include producers, consumers, and intermediaries can provide vital support for the improved use of economic evidence to inform investments in children, youth, and families. Together these stakeholders can play a more impactful role not simply by gathering but also by working together to build a sustainable, coordinated infrastructure that will support the systematic use of high-quality economic evidence. However, investments are vitally needed to help build such an infrastructure. Funders, policy makers, program developers, program evaluators, and publishers engaged in science communication each have unique opportunities to help achieve this advancement, but those opportunities, in turn, will depend in no small part on the incentives offered by the various stakeholders to each other.

Given the crucial need to improve communication among and between stakeholders, sometimes even at the most basic level, the committee identified key messages that producers and consumers of economic evidence

___________________

³ Intermediaries are defined as stakeholders who use economic evidence to enhance practice and policy through advocacy, technical assistance, or other avenues.

would like each other to know and take account of before, during, and after the production of economic evidence (see Box S-2).

RECOMMENDATIONS

Based on their research conclusions, the committee formulated recommendations for producing high-quality economic evidence; improving the utility and use of evidence; and actualizing those improvements to better inform investments for children, youth, and families.

Producing High-Quality Economic Evidence

The committee developed a set of best practices to help current and would-be producers of economic evidence understand when an intervention is sufficiently ready for an economic evaluation and what it takes to produce and report high-quality economic evidence so as to achieve transparency, consistency, and usefulness to decision makers. Although these best practices are targeted largely at the producers of evidence, they also should be helpful to consumers of the evidence, particularly with respect to assessing its quality and completeness. It is the committee’s hope that these best practices will serve as the basis for long-term improvements to support the production of clear, credible, and applicable economic evidence for decision makers. Such practices depend upon the type of economic evaluation being performed, and range from describing the purpose of an intervention, the alternative with which the intervention is compared, and the time horizon for the analysis; to valuing all the resources needed to implement and sustain the intervention; to determining the extent to which impacts are included; to employing sensitivity analysis. The list is wide ranging and fairly comprehensive and is provided in checklist form at the end of this summary for use particularly by those preparing for and engaging in an economic evaluation of an investment.

Improving the Utility and Use of Economic Evidence

To help improve the utility and use of economic evidence to inform investments for children, youth, and families, the committee developed a set of recommendations addressing the opportunities available to a diverse group of stakeholders. Public and private funders, government agencies, and education providers each hold an influential position with respect to the production and use of economic evidence. It is the committee’s hope that stakeholders will implement these recommendations to increase funding, training, and support for the improved use of economic evidence in decisions on investments for children, youth, and families.

___________________

⁴ “Funders” here might include staff in public agencies (e.g., the Centers for Disease Control and Prevention, the Institute for Education Sciences, and the National Institutes of Health), as well as staff in private, philanthropic, or other organizations.

⁵ “Funders” here might include elected officials at the local, state, or federal level; leadership of public grant-making agencies or regulatory bodies; and private funders of programs for children, youth, and families.

⁶ The key actors in “government agencies” here would include agency leadership, budget offices, and others with management and budget functions in executive and legislative branches at the federal, state, and local levels.

Actualizing Improvements in the Utility and Use of High-Quality Economic Evidence

To promote lasting improvement in the quality, utility, and use of economic evidence to inform investments for children, youth, and families, the committee determined that both producers and consumers of economic evidence need to engage at several levels beyond simply producing higher-quality and more useful evidence in a single research endeavor. Multiple stakeholder groups—including funders, policy makers, program developers, program evaluators, and publishers engaged in science communication—contribute to the production and use of economic evidence. Each of these groups can either facilitate or impede the production and use of high-quality, high-utility economic evidence. To initiate and sustain process reforms, the committee recommends that efforts be made to foster the development of multi-stakeholder collaborations and partnerships, build and fund coordinated infrastructure, and strengthen incentives for the production and use of better economic evidence.

This page intentionally left blank.

Checklist of Best Practices for Producing High-Quality Economic Evidence

For All Economic Evaluation Methods, Report the Following:

• — Specify the intervention for the economic evaluation, including a description of the intervention’s purpose, its intended recipients, the intensity and duration of services provided, the approach to implementation, the causal mechanisms, and the intended impact(s).
• — Specify the context in which the intervention was or will be implemented, such as characteristics of the population served; the time, place, and scale of implementation; and other relevant contextual factors.
• — Specify the counterfactual condition, including whether the alternative is no intervention, an alternative intervention, or business as usual. In the case of cost-effectiveness analysis (CEA) and benefit-cost analysis (BCA), ensure that the same counterfactual applies to the cost analysis (CA) and the impacts used for the CEA or BCA.
• — Determine the scope of the economic evaluation, including the type of method to be used and the perspective (and any subperspectives) for the analysis; if the societal perspective is not adopted, discuss limitations of the evidence and/or generate results from the societal perspective in a sensitivity analysis.
• — Determine the currency and reference year for all monetary values.
• — If new taxes will be used to fund the intervention, determine the assumed deadweight loss parameter. If a 0 percent rate is selected (i.e., no deadweight loss), generate results in a sensitivity analysis using loss parameters greater than 0 when accounting for new revenue required to pay for an intervention or for impacts on taxes paid or transfer payments.
• — Determine the time horizon for the analysis, and when costs or outcomes accrue over multiple years, the base case discount rate and age or point in time to which to discount (e.g., start of the intervention or a standardized child age). If a 3 percent discount rate is not selected, generate results using a 3 percent discount rate in a sensitivity analysis.
• — Determine the method for addressing uncertainty, and apply it to generate standard errors and confidence intervals for all summary measures, such as estimates of total (present-discounted-value [PDV]) costs, total (PDV) benefits, net (PDV) benefits, cost-effectiveness and benefit-cost ratios, and internal rate of return.
• — Employ sensitivity analyses to test the robustness of estimates under a variety of assumptions, including alternative discount rates,

• deadweight loss parameters, and estimates of the societal perspective if not the main perspective.

• — Determine whether equity issues need to be addressed.
• — Follow the reporting guidelines on the checklist for best practices for reporting economic evidence below.

For CA

Core Practices:

• — Value all resources needed to implement the intervention, including infrastructure needs.
• — Use shadow prices to derive an accurate estimate of the value of a resource when a market price is not available.
• — Allocate overhead costs based on use.
• — Annuitize capital investments.
• — Calculate total costs and cost components: fixed, variable, and marginal costs.
• — Calculate unit costs (e.g., cost per participant) to facilitate implementation and replication.

Advancing Practices (all core practices plus the following):

• — Prospectively plan for cost analyses to be integrated into program evaluation.
• — Use micro costing procedures whenever possible to improve the quality of intervention cost estimates and facilitate implementation and replication.
• — Define major intervention activities and identify costs associated with each, including who bears those costs.
• — Estimate costs for intervention planning, development, and adoption separately from those for intervention implementation.
• — Use Monte Carlo methods to evaluate simultaneously the implications of multiple sources of uncertainty.
• — Develop or modify budgetary and other management information systems to include relevant cost categories.

For CEA and Related Methods (in addition to best practices for CA)

Core Practices:

• — Determine an explicit rationale for including intervention impacts in the CEA and selecting the focal impact that will not be valued in the monetary unit. All included impacts should be attributable to the intervention’s theory of change. When available and relevant to the evaluation question(s), use information from well-conducted

• systematic reviews and/or meta-analyses to inform intervention impact estimates.

• — Determine whether the CEA will use a quality-of-life measure (e.g., quality-adjusted life years, disability-adjusted life years) as the focal impact and what method will be used for scoring that measure.
• — Determine whether the CEA will be limited to direct, observable economic impacts, or linked or projected impacts also will be included.
• — For impacts valued in the monetary unit (if any), use willingness-to-pay methods to calculate their prices. This may mean using a combination of market prices and shadow prices.
• — Calculate the average cost-effectiveness ratio and, where feasible, the incremental cost-effectiveness ratio.

Advancing Practices (all core practices plus the following):

• — Conduct CEA only when an intervention has been evaluated using research designs that can produce unbiased causal estimates of impact.
• — Conduct CEA from a societal perspective to produce the most comprehensive economic estimates.
• — Link or project observed outcomes only when strong causal evidence of the assumed relationship exists.
• — Estimate costs and benefits separately by perspective (e.g., participant, agency, government, other beneficiary) and by category (e.g., income, crime, health care).
• — Use Monte Carlo methods to evaluate simultaneously the implications of multiple sources of uncertainty.

For BCA and Related Methods (in addition to best practices for CA)

Core Practices:

• — Determine an explicit rationale for including intervention impacts in the BCA. All included impacts should be attributable to the intervention’s theory of change. When available and relevant to the evaluation question(s), use information from well-conducted systematic reviews and/or meta-analyses to inform intervention impact estimates.
• — Determine whether the BCA will be limited to direct, observable economic impacts, or linked or projected impacts also will be included.
• — Determine whether the BCA will include intangible as well as tangible economic impacts.

• — Use willingness-to-pay methods to calculate prices for impacts. This may mean using a combination of market and shadow prices.
• — Estimate linked or projected economic impacts using the strongest available theoretical and empirical literature. When available, use information from well-conducted systematic reviews and/or meta-analyses to inform estimates used for linking and projections.
• — Calculate PDV costs, benefits, and net benefits (total and unit). Where relevant, also calculate benefit-cost ratio, return on investment, and internal rate of return.
• — When there is concern that impact estimates may be biased (e.g., nonexperimental design, quasi-experimental design), test the robustness of findings to variation in effect size.

Advancing Practices (all core practices plus the following):

• — Conduct BCA only when an intervention has been evaluated using research designs that can produce unbiased causal estimates of impact.
• — Conduct BCA from a societal perspective to produce the most comprehensive economic estimates.
• — Link or project observed outcomes only when strong causal evidence of the assumed relationship exists.
• — Generate tangible and intangible values separately.
• — Estimate costs and benefits separately by perspective (e.g., participant, agency, government, other beneficiary) and by category (e.g., income, crime, health care).
• — Use Monte Carlo methods to evaluate simultaneously the implications of multiple sources of uncertainty.

Checklist of Best Practices for Reporting Economic Evidence

For All Economic Evaluation Methods, Report the Following:

• — The features of the intervention analyzed (e.g., logic model, intended recipients, intensity and duration of services, implementation, and other intervention features)
• — The context in which the intervention was or will be implemented (e.g., population served; time, place, and scale of operation)
• — The counterfactual (baseline or status quo) with which the intervention is compared
• — The perspective for the analysis and any subperspectives examined, with associated results
• — The currency and reference year for all monetary values
• — The assumed deadweight loss parameter, if one was used
• — The horizon for measuring economic values and, when discounting is used, the discount rate and time (or age) to which discounted
• — Summary measures of the economic evaluation results (see below for each specific method)
• — When relevant, results disaggregated by stakeholder
• — The approach for addressing uncertainty, details on how the method was implemented, and the associated standard errors or confidence intervals for all summary measures
• — Sensitivity analyses performed and associated results*
• — When relevant, any equity considerations

For cost-effectiveness analysis (CEA), benefit-cost analysis (BCA), and Related Methods That Employ Impact Estimates Also Report:

• — The evaluation method, the intervention impacts* and their statistical significance,* potential biases in estimates of causal effects, and any adjustments to estimated intervention impacts
• — All limitations resulting from the strength of the evidence of causal intervention impacts

In Addition to the Elements for All Methods, for Cost Analysis (CA) and the CA Component of a CEA or BCA Also Report:

• — The costing method (e.g., micro costing)
• — The inventory of resources used and those that are valued versus not valued in the CA

• — The method for obtaining information on how much of each resource is used, any related assumptions made, and how much of each resource is used
• — The method for obtaining unit costs, prices, or shadow prices for each type of resource; any related assumptions made; and the resulting values*

CA Results

• — Total costs and unit cost (e.g., cost per participant)
• — Fixed, variable, and marginal costs
• — The implications of methods (e.g., omission of resources, prices applied) for under- or overestimating intervention costs

In Addition to the Elements for All Methods and for CA, for a CEA Also Report:

• — Which impacts measured in the evaluation are valued in the CEA and which are not*
• — Which impacts are observed versus linked or projected, for whom they are linked or projected, and the linking or projection method — For the impacts valued in the monetary unit (if any), the prices used,* their derivation, and the geographic or jurisdictional boundary to which the valuations apply*
• — If the focal impact is a quality-of-life measure (e.g., quality-adjusted life years, disability-adjusted life years), how that measure was scored

CEA Results

• — The average cost-effectiveness ratio and, where feasible, the incremental cost-effectiveness ratio
• — The implications of methods (e.g., omission of resources in CA, prices applied in CA, causal evidence on outcomes, linkages or projections of outcomes, valuation for outcomes) for under- or overestimating cost-effectiveness

In Addition to the Elements for All Methods and for CA, for a BCA Also Report:

• — Which impacts measured in the evaluation are valued in the BCA and which are not*
• — Which impacts are observed versus linked or projected, for whom they are linked or projected, and the linking or projection method

• — For each impact valued, the price or shadow price used,* its derivation, and the geographic or jurisdictional boundary to which the valuation applies*

BCA Results

• — PDV societal costs, benefits, and net benefits
• — Benefit-cost ratio, return on investment, and/or internal rate of return
• — The PDV benefits (or costs) of each outcome valued,* with disaggregation by outcomes observed versus projected and, where possible and relevant, by tangible versus intangible benefits (e.g., for crime or child abuse and neglect)
• — The implications of methods (e.g., omission of resources in CA, prices applied in CA, causal evidence on outcomes, exclusion of outcomes, linkages or projections of outcomes, valuation for outcomes) for under- or overestimating intervention net benefits

NOTE: An asterisk denotes reporting that may be suitable for a table.

This page intentionally left blank.