3

Data Evaluation and
Software Development

The computational and the value submodels were developed in parallel and then integrated over a software platform that allows users to interact with and understand the relationships between the model input and output. The model development and interface development occurred concurrently. The committee received and adjusted its software development strategy based on feedback received from consultant concept evaluators.

In the following sections we describe the selection of vaccine candidates and of the related data to be fed into the model and then the actual model development and evaluation process.

Selection of vaccine candidates

The committee considered several hypothetical vaccine candidates from the perspectives of the United States and of a developing country. The committee agreed on South Africa as the particular developing country for this process since its income profile, its population, and its health, economic, and social priorities are vastly different from those of the United States. A second reason for selecting South Africa was the availability of input data for disease burden and vaccine estimates, which were necessary to populate and test the model.

The five hypothetical candidate vaccines chosen were a universal influenza vaccine plus vaccines against tuberculosis, group B streptococcus, malaria, and rotavirus. However, as the work of assembling the data for the first vaccines began, it became clear that the present scope of work

For example, the universal influenza vaccine addresses a disease that is important in both high- and low-income countries, and the convenience of a single vaccine for all influenza strains would make it readily useful for all parts of the world. Furthermore, influenza affects all age groups and causes widespread morbidity worldwide. In contrast, tuberculosis does not pose a significant threat in high-income nations, thus a vaccine for tuberculosis would likely be of most use in the low- and middle-income countries. And group B streptococcus vaccine would be pertinent for both low- and high-income countries but is designed for administration to pregnant women (a special population) and would confer benefits to their infants. Additional information on the impact of influenza, tuberculosis, and group B streptococcus can be found in Boxes B-1, B-2, and B-3 in Appendix B.

Data sourcing and analysis

In its data-gathering process, the committee did not attempt to develop the best or most detailed estimates about each disease. The objective was instead to obtain reasonable data that could help the committee evaluate the model rather than to generate precise projections about specific vaccines.

The committee chose to develop reasonable estimates for data based on literature reviews and expert opinion, and it sometimes also relied upon committee-generated assumptions because much of the information required for the model, especially information concerning South Africa, was not available. It is thus reasonable to view the data inputs as characterizing hypothetical vaccines against influenza-like, tuberculosis-like, and group B streptococcus–like syndromes.

The estimates and assumptions used in this model were based upon literature reviews, publicly available data provided by international agencies such as the World Health Organization (WHO), and publications of various other organizations, such as the Agency for Healthcare Research and Quality (AHRQ) and the Healthcare Cost and Utilization Project (HCUP) in the United States.

For each candidate vaccine, the model used several categories of inputs (see Table 3-1 for specifics):

•Health characteristics, including disability-adjusted life years (DALYs) and quality-adjusted life years (QALYs), were obtained from the available literature. DALYs were calculated by assigning DALY weights from the Global Burden of Disease study (Mathers et al., 2006). Similarly, HUI-2 was used as a measure to calculate QALYs. When the exact condition of concern was not categorized in DALY and HUI-2 weights, proxies were used. Appendix C provides a listing of the data used in the model.

•Vaccine characteristics, including the number of years to full adoption, population coverage rate, effectiveness, length of immunity, doses required per person, costs of administration, and research and development costs. Vaccine traits were a combination of factual data and expert panel judgments. Vaccine efficacy, vaccine-associated complications, coverage, and the number of doses required for immunity were estimated from the literature, whereas time to adopt a vaccine within an immunization scheme, development risk, and innovation for new delivery methods were guided by expert opinions. Data on health care costs for disease and vaccine candidates were obtained from both a literature review and governmental Web sites such as those for HCUP and CDC for the United States and WHO’s Choosing Interventions that are Cost-Effective (CHOICE) project for estimates of health care services costs in South Africa.

For each of the selected vaccines, assembling the data needed for the model presented a different set of challenges.

Tuberculosis poses a significant health challenge in South Africa, and published literature concerning the magnitude of the disease is available.

SMART Vaccines submodels

SMART Vaccines includes two submodels—the computational submodel and the value submodel. As previously shown (Figure 2-1), the computational submodel calculates multiple health and economic measures associated with new vaccine candidates. Many of these measures build upon the work presented in the 2000 IOM report. The computational submodel evolved with the improvements in the health and economic attribute listing for the model. The desire for interpretable health and economic attributes drove much of the computational submodel design.

Early prototypes strongly resembled the model presented in the 2000 report. Those prototypes were tested using the same input information and were determined to reliably replicate the results of the 2000 report. However, this initial prototyping highlighted several limitations in the analytical structure of the 2000 report, specifically in the context of accommodating the following features:

• Computations for all desired health and economic attributes.

• Variations in timing between vaccine administration and onset of disease or death.

• Differences between vaccines that protected for different lengths of time (i.e., 5-year universal influenza vaccine versus 1-year seasonal influenza vaccine).

Limitations in flexibility directed the modeling efforts toward a population process model whose technical aspects are presented in Appendix A.

The computational submodel comprises seven computed attributes derived from health, vaccine, and economic inputs. The remaining 22 attributes, called “qualitative attributes,” were defined in an iterative process by the committee. After formal definitions were developed, levels of assessment were specified (Table 2-1).

The health and economic attribute measures were stratified by category (e.g., Level 2 = $/QALY between $0 and $10,000) so as to not over-specify computational model results, given the inherent uncertainty in input information. Determining the appropriate categories for health and economic measures that are to be generalized across populations of varying size, disease incidence, and mortality rates is a complex process. The categorization of the health and economic attributes needs to be conducted through a thorough evaluation of the model, supported by epidemiologic and economic evidence. This categorization has yet to be completed, but the preliminary assessment resulted in an initial set of categories to use as examples. The qualitative attributes not generated by the computational model are directly assessed by users. Definitions of categories for direct assessment were developed in an iterative process and then finalized. After finalizing the attribute definitions and assessment categories, the committee incorporated the multi-attribute weighting approach. The committee chose the rank order centroid method described in Chapter 2 for ease of use and reliability.

Development of the computational submodel

The computational submodel contains expressions for health and economic values that are based on a population process model. The process model is initialized at year i = 0 for a stationary population with: no vaccine (i.e., the baseline population); the vaccine in steady state delivery; and the vaccine first being introduced.

Annualized health and economic values are calculated by comparing a population with a vaccine in steady state to a baseline population after aging 1 year. Values capturing the efficiency of the investment (i.e., cost-effectiveness) are calculated by comparing a population where the vaccine is first introduced to a baseline population after aging 100 years. The following are further relevant details about the three types of populations:

The age-specific population process model simulates measures of population size for the total population, the target population, the vaccinated immune members of the populations, the vaccinated susceptible members, the not-vaccinated immune members (i.e., those who have indirect protection through herd immunity), and the not-vaccinated susceptible members. Simulated health measures include incident cases, deaths by disease, vaccine complications, all-cause deaths, and cause-deleted deaths. Mathematical expressions for these process measures may be found in Tables A-1 and A-2 in Appendix A.

Health and economic attributes are calculated from the population process model with mostly linear expressions (as shown in Tables A-1 and A-2) to serve as a starting point for the committee’s modeling effort. Annualized measures are differentiated over the first year i = 1 between a population with no vaccine and a population with the vaccine in steady state. These annualized measures include deaths averted, cases prevented, QALYs gained, DALYs averted, net direct costs, workforce productivity (i.e., indirect costs), and one-time costs. The length of time associated with the annualized health and economic attributes associated with death and permanent impairment is assumed to be 6 months, as this is the average time of death between year i = 0 and year i = 1. Within these tables, vaccine populations for annualized measures refer to the vaccine-in-steady-state populations.

Alternatively, calculations on cost-effectiveness measures (i.e., $/QALY or $/DALY) are performed over 100 years. Time durations incorpor-

ated within QALYs and DALYs (i.e., included in cost-effectiveness only) associated with death and permanent impairment are assumed to be future life expectancy. Life expectancy is adjusted for baseline health utility indices (i.e., HUI2) for QALYs only. Life expectancy is discounted for both QALYs and DALYs when a discount factor is introduced. Expressions for cost-effectiveness measures may be found in Tables A-1 and A-2. Within these tables, vaccine population references are assumed to be the populations where the vaccine is first introduced.

Evaluation of the computational submodel

The computational submodel has been evaluated using four base cases for preventative vaccine candidates. These cases, given in Table 3-2, are for seasonal influenza, group B streptococcus, and tuberculosis within the United States (2009) and for tuberculosis within South Africa (2009).

Table 3-2 presents input assumptions for the target population, the duration of immunity, the cost to administer, the herd immunity threshold, and coverage. It also displays annualized health and economic attribute measures applicable to a vaccine in a steady state population and efficiency measures for a population in which a vaccine is first introduced. These measures are summed over 100 years and discounted at three percent. These evaluations allow for a constructive comparison of characteristics across base cases.

The model identifies the vaccine for seasonal influenza (i.e., with 1-year duration of immunity) having the largest health impact in terms of averting deaths, preventing cases, and increasing health-adjusted life years within the United States. Direct costs are notably high because annual administration (i.e., delivery costs) to an assumed undifferentiated target population of all ages is much more expensive than delivering the vaccine solely to infants. However, given improvements in health-adjusted quality of life, the cost-effectiveness is greater for the seasonal influenza vaccine than for other candidates in the United States.

The evaluation of the base cases demonstrates major differences between targeting tuberculosis in the United States and in South Africa. The health and efficiency attribute measures are improved within the South African population, where disease incidence is much higher. In South Africa administering the vaccine in steady state is cost-saving (i.e., net direct costs <0). It is important to note that the corresponding efficiency measures do not demonstrate cost savings (i.e., cost per QALY or DALY >0). This highlights a difference between examining vaccine candidates in steady state and the standard computations of cost-effectiveness

The base cases were altered to test the relationship between the input information and the health and economic attribute measures (i.e., outputs). This verification process was performed across all inputs with example test cases shown in Table 3-3.

For example, the seasonal influenza vaccine (i.e., with 1-year duration of immunity) was altered to reflect a hypothetical universal vaccine that would provide protection for 10 years. A projected increase in the administrative costs was also included. These changes resulted in a decrease in delivery costs and improved efficiency because of less frequent administration compared to the seasonal base case. However, the improvements are substantially mitigated by the projected increases in the cost of vaccine.

Similarly, the reductions in cost to administer the group B streptococcus vaccine demonstrate more desirable economic measures. Herd immunity threshold was set at 80 percent for tuberculosis in the U.S. test case. Intuitively, the resulting health attribute measures should increase as a result of the indirect protection associated with herd immunity. Finally, the vaccine coverage for tuberculosis in the South Africa base case was increased. This resulted in proportional increases in the health and economic attribute measures. The health impact is greater; however, the cost-effectiveness remains constant compared to the base case.

Simulation of the value submodel

The committee also developed an iterative version of the value submodel in a worksheet in order to simulate and understand the variations in user preferences of attributes. Figure 3-1 displays a screenshot of the user worksheet. Two large blocks and one graph make up the screen. In the top block there are four columns; from left to right these are:

1. Two yellow columns in which the user selects and rank orders the subset of attributes to be used in the multi-attribute utility model.

2. A white area listing the 8 categories of attributes, with a total of 29 attributes.

Figure 3-2 shows a closer view of the user input areas. Five hypothetical vaccines are shown in the right-hand yellow columns. The hypothetical influenza vaccine is conceived of as a modest improvement on an existing annual influenza vaccine. Reading down the column titled Influenza,

one can see that it is rated as Level 2 on the attribute Premature Deaths Averted per Year (Attribute 1.1), as Level 1 on Incident Cases Prevented per Year (Attribute 1.2), and so forth down the column. The figure shows the levels entered for five hypothetical vaccines. Levels for each attribute are defined in the worksheet (Figure 3-1) and in Table 2_1.

The purpose of this spreadsheet was to allow the committee to experiment with the value computations part of the SMART Vaccines model. This permitted the committee to do “What if?” modeling quickly. But it also required the committee to estimate or fabricate entries for the hypothetical vaccines outside of the computations of the formal model.

Figure 3-2 shows a situation in which the user has selected 11 attributes with which to evaluate the vaccines. The submodel requires that Likelihood of Successful Licensure (Attribute 5.2) be selected in order to ensure that the user considers this factor. This requirement was added after some concept evaluators of SMART Vaccines Beta strongly endorsed

To demonstrate how perspectives might affect ranking choices, two hypothetical perspectives are presented: a vaccine producer and a health minister of a developing country. In this example, the user (say, a vaccine manufacturer) has entered numbers from 1 to 11 in the second column to indicate the rank order of importance of the selected attributes, with 1 being most important and 11 being the least important. Once the ranking is completed, the output of the value model is shown in the lower block (Figure 3-3) and the graph (Figure 3-1).

Figure 3-3 shows the tabular output from the simulated value submodel spreadsheet. The tabular output displays, from left to right, the weights associated with each of the selected attributes, the attribute labels, the subtotal percentage weight in the model assigned to each of the eight logical groupings of attributes, and the single attribute weighted score for

FIGURE 3-3 Output of the simulated value submodel.

The weights in the table are assigned using the rank order centroid method described in Chapter 2. They are displayed as percentages of the total weight in the model. If the user were to re-rank the attributes, the weights would change, as would the subscores and the total scores for the vaccines. If the user were to change the achievement levels for the hypothetical vaccines in the yellow area in Figure 3-2, these scores would change. This output table shows the detailed effects on the priority scores of the achievement levels of vaccines on the attributes, of the user’s attribute selection, and of the user’s ranking of attributes.

Now, let us assume the perspective of a health minister from an emerging South American country who may wish to place the priorities for a new vaccine in the following order:

1. QALYs Gained or DALYs Averted

2. Benefits Infants and Children

3. Reduces Challenges Relating to Cold-Chain Requirements

4. Cost-Effectiveness

5. Likelihood of Successful Licensure in 10 Years

The value experiment and scenarios

To further illustrate the effect of—and sensitivity to—different choices in the multi-attribute utility model, a simple exercise was performed, called the “value experiment.” Six vaccines, for use against one or the other of two diseases, were considered; some of the vaccines actually exist, and some were fictitious, created for use in the experiment. The value experiment was designed to test and illustrate the process of selecting values and to generate sample sets of values that would then be assigned weights and used to test the model. This experiment was conducted while the prototyping of the model was ongoing, so the vaccines evaluated and the assumptions were slightly different from the completed SMART Vaccines Beta.

The committee members and staff ranked the following six vaccines:

1. influenza with 1-year efficacy;

2. influenza with 5-year efficacy;

3. tuberculosis with 3-year efficacy;

4. tuberculosis with lifetime efficacy;

5. influenza with 50 percent increase in efficacy for those receiving vaccination; and

6. tuberculosis with 3-year efficacy, but with a 100-fold increase in incidence in the population risk.

Participants were asked to think about the candidate vaccines for the United States and for low-income countries. Half the participants ranked these vaccine characteristics from the perspective of the United States and half from the perspective of low-income countries. They were provided with a list of an earlier draft attributes deemed important for vaccine prioritization (see Table 2-1). The participants then selected up to five attributes of highest importance to them and ranked them based on their own perspectives. The group included individuals with diverse perspectives, from infectious disease epidemiologists and authorities on health care in low-income countries to experts in health economics, systems engineering, and decision sciences.

The weights were then applied against the vaccines under consideration through SMART Vaccines Beta to calculate each person’s priority score for the vaccine and disease combinations. The participants ranked the priorities in the following order:

1. Premature deaths averted per year

2. Incident cases prevented per year

3. Likelihood of successful licensure in 10 years

4. QALYs gained or DALYs averted (DALYs were #4, and QALYs were #6)

5. Cost-effectiveness ($/QALYs)

This experiment shows how weights can have a significant effect on the priority scores. Table 3-4 shows the results for participants A through N. Compare, for example, rankings produced by the multi-attribute utility weights of persons A and B. The weights specified by A led to the highest value (priority score) being placed on the influenza vaccines, the order being quite understandable intuitively (highest for a 1-year with 50 percent increase in efficacy, then the 5-year vaccine, followed by the 1-year vaccine), with the two tuberculosis vaccines running considerably behind. The weights specified by B, however, gave quite a different ranking. That participant gave the highest value to the hypothetical tuberculosis vaccine with 3-year immunity in a population with 100 times higher incidence.

Two other observations emerge from examining this table. First, one cannot meaningfully compare numerical scores across different raters. Person B’s score of 44 (tuberculosis with lifetime immunity) has no relationship to person K’s score of 44 (influenza with 5-year immunity). To equate these two scores would be to make a mistake similar to saying that 65ºF and 65ºC were the same temperature. To compare values across similar or different vaccine scenarios for two persons is faulty because the priority

This raises the issue of the meaning of the scale of utility in the multi-attribute utility value models. The issue, as discussed in Chapter 2, is the same as encountered when using a temperature scale such as fahrenheit or centigrade; one may be tempted to say that one temperature is “twice as warm” as another, but 20ºC is not twice as warm as 10ºC, nor is 90ºF twice as warm as 45ºF. Using the temperature analogy, the multi-attribute value function does not begin at “absolute zero,” and hence 80 is not twice as high as 40.

What do the scores mean? The most important result they offer is to provide rank orderings. Furthermore, for a given person, it is generally the case that differences in scores have meaning, so that the difference between 20 and 40 in a priority score has the same meaning as a difference between 50ºF and 70ºF. This is just the same as saying that the difference between 50ºF and 70ºF is the same as the difference between 20ºF and 40ºF—both represent a difference of 20ºF, nothing more and nothing less.

Since each rating process is unique, based on the user’s perspectives, different rankings will produce different values for users. A multinational health organization will have different weights and priorities than a minister of health or finance from a developing country. The Pan American Health Organization, for example, may choose values that lead to a rank for tuberculosis higher than for influenza, whereas a vaccine manufacturer might have different weights, thus producing different rankings. Similarly, a basic research institution such as the National Institutes of Health (NIH) may have different priorities than a private company. A manufacturer is likely to be interested in return on investment and securing a market for its product, therefore assigning different values than an NIH official who may be interested in scientific advancement that may also apply to other vaccines. The flexibility to accommodate multiple priorities from many users is an asset, as it allows the many sectors within the vaccine enterprise to enlist in the broader discussion that is important in vaccine development and prioritization.

In short, SMART Vaccines allows users to rank attributes to reflect their value preferences. Once users have ranked the set of attributes for the

Software development: Operational features of SMART Vaccines Beta

This section of the report presents screenshots of SMART Vaccines Beta to illustrate its current features. The intention is to provide readers with an understanding of the conceptual flow of the model in a software platform, the data needs of the program, and the current user interface and navigation, which has potential for further improvement. SMART Vaccines Beta was developed using three software tools: MATLAB for algorithm development and testing, JAVA Servlets for the middleware, and Axure for visual prototyping and interface development. The preliminary database was managed using Microsoft SQL Server.

In its current version SMART Vaccines Beta has a six-step process for producing a value score for vaccine candidates. All the data and the results shown in the screenshots are hypothetical and should not be interpreted as any form of endorsement by the committee or the Institute of Medicine.

Step 0: Terms of Agreement

Figure 3-4 shows the disclaimer page (Step 0) of the software, which requires the user’s agreement to the terms. All of a user’s work can be saved for future use and modified and re-saved to allow variants in baseline data conditions. User accounts with password protection are possible options.

Step 1: Values

The values page (Figure 3-5) introduces the user to the overall program structure and navigation. Every page has a similar panel of six tabs at the top (with this page having “Values” highlighted). Typically users will move sequentially through these tabs. Every page the user visits will have, in the upper right corner, a pop-up window for a glossary of terms and instructions. The advanced mode (presented in all the screenshots) allows the user to enter data in order to consider attributes beyond health and economic factors. The basic mode, by contrast, has options relating only to health and economic attributes in case the user has a predetermined attribute (say,

SMART Vaccines Beta converts the rank order of attributes selected in the drag-and-drop box into numerical weights to be used in the multi-attribute value model. Chapter 2 described this process and provides references to justify this approach. Later versions of SMART Vaccines should be able to incorporate the direct entry of value weights by the user.

Step 2: Demographics

Figure 3-6 shows the demographics screen in which the user enters population data. The user can either pull up pre-specified populations (such as those of the United States or South Africa in the lower left panel) or begin with a blank template and fill in data for an entirely new population. The

Step 3: Disease Burden

Step 3 takes the reader into the specification of disease burden (Figure 3-7). Unlike Steps 1 and 2, Step 3 must be filled out separately for each disease that might be prevented by a candidate vaccine (e.g., influenza, tuberculosis, or group B streptococcus), and, as shown below, subsequent screens apply to each candidate vaccine, and there may be more than one candidate vaccine per disease.

In the Economic tab of Step 3 (Figure 3-8), users specify typical treatment patterns for each disease in question and the costs of each type of treat-

impairment as appropriate for each disease. These categories of morbidity are combined with the cost of each type of treatment (see bottom of Step 3 screen) to estimate the costs of treating unprevented disease. Figure 3-9 shows the lower half of this screen, using the scroll bar on the right side of the screen. This feature is common to most pages of SMART Vaccines.

FIGURE 3-12
The SMART Vaccines Beta Vaccines (complications) screen asks the user to enter information relating to expected vaccine-induced complications, permanent impairments, and deaths.

Step 4: Vaccines

Population

In Step 4 users enter vaccine-specific data (Figure 3-10). Each potential disease under consideration (in this example, influenza, tuberculosis, or

In this step the user specifies the potential attributes of a specific vaccine (Figure 3-11). Of course, these are not known with certainty before actual development, so users must use expert opinion to conjecture about the candidate vaccines. These attributes are central to the issues of vaccine prioritization because they include basic aspects of the vaccine (e.g., how many doses and costs per dose to purchase and administer), research and development costs, licensing costs, and expected time to adoption. The

Step 4 also includes an entry screen for potential complications that a new vaccine may cause (Figure 3-12). These data are similar in concept to those

in Step 3 (Disease Burden), but in this case they refer to complications of a candidate vaccine rather than to the consequences of unprevented disease. Users need to specify each possible complication and all associated data. Since these complications are unknown until a vaccine is fully field tested (or used widely so as to detect rare complications), users will necessarily draw on expert opinion and work by analogy from vaccines with similar characteristics (e.g., live or inactivated virus or types of adjuvants). Figure 3-13 shows the bottom of the Complication page using the scroll bar at the screen’s right side.

Step 5: Value Assessment

Step 5 asks users to enter qualitative information about each vaccine. These come in eight categories, as previously shown in Table 2-1. Each one of these categories opens up like an accordion menu to show all of the qualitative attributes associated with any vaccine, whereupon the user checks the appropriate category for each attribute. Each category has a pop-up bubble associated with it to describe to the user the committee’s intent or definition regarding a particular categorical choice for each attribute (each indicated by a symbol). The user need not fill out these data queries if the attributes in question have not been selected in the value choices (Step 1). Figure 3-14 shows this step with the Health Considerations bar opened up,

and Figure 3-15 shows the same step with the Economic Considerations bar opened up.

Step 6: Value Assessment and Score

The screen at Step 6 shows values for all of the calculated attributes for each vaccine under consideration (Figure 3-16). This provides a single “dashboard” point that shows what all of the previous data entries lead to in calculated attributes. For example, Premature Deaths Averted per Year uses data on population size by age, disease incidence by age, vaccination rate by age, vaccine efficacy rate by age, and the case mortality rate to compute the number of premature deaths averted per year. A similar computation creates the Incident Cases Prevented per Year. Calculation of QALYs gained and DALYs averted also include information (entered at Step 2) regarding disease burden.

As noted before, users may select either DALY or QALY measures, but not both. If a user selects the DALY measure, he or she has the option (at the upper left of the Step 6 screen) to use or avoid the associated age-weights. The calculated illustrative value scores are shown in Figure 3-17.

Consideration of uncertainty

In this phase, the committee was unable to explicitly model issues relating to uncertainty in SMART Vaccines Beta. In Phase II the committee will consider various elements of uncertainty to be included in SMART Vaccines 1.0. Sources of uncertainty and how they affect SMART Vaccines are briefly discussed, along with some possible methods to address these issues in Phase II.

Uncertainty About the Likelihood of Successful Licensure

SMART Vaccines Beta includes one uncertainty component but instead of listing it as a probability the committee characterized it as a value attribute: “Likelihood of Successful Licensure in 10 Years” under “Scientific and Business Considerations” (Table 2-1). The uncertainty related to the time the vaccine may become available for public use affects judgments about priority.

Otherwise, some possible ways to address the issue of uncertainty include programming the uncertainty component into the computational submodel as a delay between “now”—the time when the priorities are being set—and the time when the health benefits due to vaccination might

Other Uncertainties

Manning and colleagues (1996) identify three sources of uncertainty in cost-effectiveness models (that otherwise affect any computational model such as SMART Vaccines): (1) parameter uncertainty; (2) model structure uncertainty; and (3) model process uncertainty.

Parameter Uncertainty

The computational submodel in SMART Vaccines Beta, although simplistic in its current form, is a function of many parameters: population modeling, estimates of health burden and benefits, and estimates of health care costs. Each of these parameters has components of uncertainty surrounding it.

The current model does not incorporate uncertainty about these parameters in its computations. The most straightforward method to do so would be to specify a distribution surrounding each parameter and then use Monte Carlo simulation to sample from the distributions and compute results for each sample. Then a distribution for each of the computational outputs could be built, and these, in turn, could be used to determine an overall distribution on the priority score.

The committee elected not to do this for SMART Vaccines Beta due to two concerns. The first relates to the source of the distributions for input parameters. Some parameters may affect all vaccine candidates, such as population life tables, while others are specific to an attribute or a vaccine candidate. It is well known that life tables are built from population sample data and thus have uncertainty concerning every age-specific mortality rate or life expectancy. Whether these uncertainties should be incorporated in the computational submodel is an open question; many models such as these take population and life-table values as “given” without incorporating any uncertainty surrounding them. In any case, with additional effort, these uncertainties could be represented in the computational submodel.

Current capability for sensitivity analysis

SMART Vaccines Beta has the capability to permit variations in attributes to observe the consequences in the final utility score. This sensitivity analysis can be conducted manually in the current version, and indeed, differ-

Beta concept evaluation

Following the development of SMART Vaccines Beta, a concept evaluation session was organized to obtain feedback from potential users. Each of the 11 consultant evaluators participated in a webinar led by a committee member and staff; four similar webinars were held, with two to four evaluators participating in each session. The evaluators were asked to provide feedback regarding the basic concept, software design, technical features, potential applications, and audiences. In general, the overall concept of SMART Vaccines Beta was received positively, even enthusiastically, with the exception of one evaluator who shared concerns regarding the basis and extension of the work. Many of the features of SMART Vaccines Beta have already been updated in response to the comments from concept evaluators. More important, many features have the potential to be upgraded in Phase II of this study.

The committee’s observations and views on the next steps in the enhancement of SMART Vaccines Beta are presented in Chapter 4.