5

Programs or Principles?

A prominent question in the prevention field has been whether it is possible to identify similar component elements in evidence-based programs and disseminate those elements, whether identified as principles, practices, or kernels, rather than the programs themselves. This question in turn generates a number of more detailed questions: What needs to be in place or what are the criteria for determining that a principle, practice, or kernel

is effective and ready for dissemination? How can implementation fidelity and outcomes be measured for different kinds of interventions? A panel of five experts examined these and other questions both from the standpoint of researchers striving to achieve a common vocabulary and agreed-upon methods and from the standpoint of communities working with researchers.

EVALUATING THE EFFECTS OF PROGRAMS AND PRINCIPLES

Stephen Gies, senior researcher at Development Services Group, began with a set of definitions for programs, practices, and principles.

A program is a planned, coordinated, and prescribed group of activities and processes designed to achieve a specific purpose, he said. Generally, programs are based on models that are supposed to be implemented as designed.

A practice he defined as a general set of programs, strategies, or procedures that share similar characteristics with regard to the issues they address and how they address them. For example, mentoring can be seen as a practice, as can the idea of a drug court. Practices “follow very similar ideas but can often be implemented in different ways,” he said.

A principle—which Gies equated with a component, kernel, or active ingredient—is a behavior-influenced procedure shown through experimental analysis to affect a specific behavior and that is indivisible in the sense that removing any of its components would render it inert. This definition is based on the ideas of Embry and Biglan (2008), but Gies also emphasized the role of the inactive ingredients in a principle, since sometimes these inactive ingredients have unexpected and surprising effects on the active ingredients.

Given these definitions, Gies presented a way of evaluating effectiveness within programs and across programs (see Table 5-1). If different programs share some but not all principles, the effectiveness of the programs or of the

TABLE 5-1 Comparisons Within and Across Programs

SOURCE: Gies (2106). Available: http://sites.nationalacademies.org/cs/groups/dbassesite/documents/webpage/dbasse_172961.pdf [May 2017].

component principles can be measured. Furthermore, the effectiveness of programs or principles can be compared through meta-analyses and moderator analyses. A meta-analysis can determine the effect size of a program for a particular outcome. A moderator analysis can determine the effect size for every program that has a particular component.

This analytic approach has both strengths and weaknesses, Gies observed. Among its strengths, it integrates similar programs, multiple versions of programs, and adaptations of those programs, since the components of different programs can be analyzed. It summarizes evidence into a single statistic (generally effect size) while estimating the actual magnitude of this effect. It also increases power and precision by pooling different studies.

Weaknesses include the heterogeneity of programs and principles, the possibility of selection bias, the need for “uncommon” statistical expertise, difficulties of interpretation, and the use of point estimates rather than growth curves or some other output. Nevertheless, programs, practices, and principles in prevention science can provide valuable information for researchers as well as communities considering different interventions.

STUDYING THE EFFECTS OF PRINCIPLES DERIVED FROM PROGRAMS

Patrick Tolan, professor of education and of psychiatry and neurobehavioral sciences at the University of Virginia, used slightly different definitions of programs, practices, and principles. As did Gies, he defined programs as full interventions with multiple activities. They typically involve long time periods and substantial investments.

Practices he defined as the seminal components that are extracted from multiple programs. As Tolan put it, the assumption is that practices “must work because they’re in multiple programs.”

Principles he defined as activities, processes, or relational orientations that underlie programs and are expressed by practices. He associated principles with the theories underlying a program, commenting, “What do you think you’re doing, and what is it about these programs that reflects similar theory?” In that sense, principles act as guiding ideas toward which practices and programs should be moving.

Tolan also defined a set of standards that should be the same across programs, practices, and principles. Trials should be implemented well and without bias, the outcomes should be positive, replication and generalization should be possible, and the implementation requirements should be clear. These standards are a prerequisite to valid scientific inference, he said.

With this context, he focused his presentation largely on principles, which he said have not been investigated carefully. They have both implementation requirements—what people are asked to do—and engagement

components, which are shaped by relationships, motivations, and other factors. Both of these components are factors in going to scale, said Tolan, since they both affect what is required of a practice. At the same time, they both reflect theories of change and behavior.

As described by Gies, mediator analyses can help reveal the nonreducible kernels in a program. Tolan also advocated study of the resistance to non-fidelity. “We have to work in low-fidelity systems,” he insisted. “There’s turnover. We don’t pay these people very much. . . . So how do you test interventions that can work in those conditions?” Some studies of child welfare have assumed that the systems have low fidelity because people who provide good child welfare practice get moved into administration or get burned out and leave. “How can we start to think about that, for example, as the conditions of going to scale?” he queried.

In addition, Tolan called attention to recent mediational analysis work on testing not only how something works, but also how it does not work. For example, a counterfactual explanation or something assumed not to change in a given intervention that could explain an outcome could also be assessed. Also, considering multiple and cascading mediators as processes of interventions are “wonderful ways for us to start to get inside the black box,” said Tolan, by helping to reveal the key parts of prevention that make a difference.

Tolan concluded with potential areas of study for this type of research. The first is to use benchmark approaches (critical features of effective interventions) to determine if greater use of prescribed activities leads to better programs. The second is to look at mediation more carefully and in a more complex manner, as described above. The third is to examine the personalization of effects—for example, where an intervention works better for girls than for boys, or where an intervention has very different effects on different children. The fourth is to disassemble interventions to optimize their components and determine the minimal intervention needed for effects. The last is to study whether excellent consulting and technical aid to home-grown programs that people want to implement produces better results than well-implemented and supported prescriptive programs. “We’ve never tested what happens if somebody came in and said, ‘Let me give you the best advice we can on how to make your program work well,’ versus ‘Here’s how you do my program well.’ We need to do that, [because] if technical assistance is what makes programs effective, we have a huge potential in terms of impact at a much smaller price than mounting specific programs with extensive training requirements,” he concluded.

DISTINCTIONS BETWEEN PRINCIPLES AND KERNELS

According to Anthony Biglan, senior scientist at the Oregon Research Institute, most of the components of evidence-based family and school

interventions were initially developed through single-case experimental designs, which tested the relationship between environmental influences and behavior. These single-case designs may have relied on A-B-A designs, multiple baseline designs, or multi-element designs, but together they have revealed a great deal. For example, Biglan said, work during the past 35 years has resulted in “a careful understanding of the relationships, the interactions, the moment-to-moment interactions between a parent and a child, and the ways in which they affect the development of the child and the ways in which those things can be changed.”

Randomized trials have taken center stage in recent years. These trials allow tests of whether an intervention is sufficiently replicable across cases that it produces results superior to comparison conditions. For example, parents can be taught an action that changes a child’s behavior, as demonstrated by a trial.

Biglan described an evidence-based kernel as a simple behavior-influence technique that has been extensively validated, mostly through single-case experimental designs (Embry and Biglan, 2008). Kernels are generally simple and indivisible, so that removing any part of them makes them inactive. They produce quick, easily measured change and are the active ingredients of evidence-based programs.

Biglan went on to identify four types of kernels:

1. Antecedent kernels happen before the behavior. A school-based example is a drawing of feet next to a line in an elementary school classroom where students will line up. Another is a technique Biglan called “Beat the Timer,” where if a child can get ready for school within 15 minutes, a parent will read a book with the child for 5 minutes.
2. A reinforcement kernel happens after the behavior. An example is a reward offered in the Good Behavior Game.¹ Another example is a note of praise that a child can take home after a prosocial behavior.
3. A physiological kernel changes the biochemistry of behavior. An example that has been evaluated in randomized trials is an omega-3 fatty acid supplement.
4. A relational frame kernel creates verbal relations for a behavior. Biglan described a teacher who asked his students what they would see, hear, feel, and do in a wonderful classroom. The teacher referred to positive behaviors as pax and negative behaviors as spleems. In playing the Good Behavior Game, if student teams had three or fewer spleems, they would get a reward.

___________________

¹ For more information about the Good Behavior Game, see http://goodbehaviorgame.org [May 2017].

By definition, kernels have been validated, and their impact or lack of impact on behaviors is in most cases immediately observable. However, Biglan pointed out that the implementation of any practice or program needs to be accompanied by monitoring of its impact, just as the treatment of hypertension needs to be accompanied by the measurement of blood pressure. It is not enough to say, “It was researched over here, so we’re pretty sure it will work there,” he stressed.

Biglan also mentioned two overarching or higher-level principles. The first is the need to minimize toxic social conditions. Reducing the prevalence of toxic social environments is probably the most generic and most important public health challenge that exists, he said. “If the only public health principle we chose to pursue was an increase in the longevity of every member of society, we would quickly conclude that reducing the prevalence of toxic social environments was the most important means of achieving that goal, and so we need to reduce those in childhood,” he stated.

The second overarching principle is the need to replace aversive means of control with more positively reinforcing practices. Public policies that reduce poverty and inequality and reduce the use of punishment in schools and the use of punishment in criminal justice systems are vital to improving the well-being of children. “The public health goal needs to be to increase the prevalence of nurturing families,” Biglan said.

REBUILDING THE EVIDENCE BASE FOR PROGRAMS

Initiatives that include collections of programs also can provide insights into distinctions between programs and their components. For example, the Office of Adolescent Health’s Teen Pregnancy Prevention (TPP) Program is a two-tiered evidence initiative, funded at a level of about $70 million annually, that is intended to replicate programs proven effective through rigorous evaluation. A smaller amount of funding, about $24 million a year, is set aside for developing and testing new and innovative approaches. “The evidence isn’t static,” said Amy Margolis, director of the Division of Program Development and Operations in the U.S. Office of Adolescent Health. “We need to continue to test and develop new approaches.”

TPP is focused on reaching the communities with the greatest need and the most vulnerable youth within those communities. The first cohort funded 102 grantees from 2010 to 2015 and served 500,000 youth in 39 states and the District of Columbia. The second cohort, begun in 2015 and scheduled to last through June 2020, encompasses 84 grantees serving 1.2 million youth. Among 500,000 youth served in the first cohort: 74 percent were ages 14 or younger and 18 percent were ages 15 to 16, while 37 percent were Latino, 30 percent black, and 23 percent white. Ninety-five percent of all sessions were implemented as intended, and 92 percent of all

sessions were implemented with high quality. TPP allows minor adaptations to interventions, but it discourages major adaptations. Youth attendance was high—on average, youth attended 86 percent of all sessions. The program has resulted in 66 manuscripts published and nearly 1,300 national, regional, and state presentations delivered.

TPP has invested significantly in evaluation. During its first 5 years, it supported 41 independent rigorous evaluations through a mix of contracts and grants. For example, one contract evaluated three different evidence-based programs in three different sites to examine the differences among sites. Sixteen different grantee-led evaluations were all designed to meet the program’s standards for a moderate or high rating for evidence review. Altogether, 19 evaluations of 10 different evidence-based programs provided information about where, when, and with whom programs are most effective. These evaluations found, for example, that 4 of the 10 evidence-based programs were effective in new settings and with new populations. “We need to know where the programs are effective and where we’re not seeing the same sorts of effectiveness, because that helps communities decide which programs are going to be the best fit for them,” said Margolis.

Margolis drew several lessons from TPP’s history. First, programs need a body of evidence, not just a single evaluation. “We need to start with the evidence that we have, but we should continue to build on that body of evidence,” she said. Independent replication evaluations are critical, again to build the body of evidence that supports a program.

Some programs can be generalized to a wide variety of groups with the same results, but other programs work best with more targeted settings and populations. Programs that were effective at one time may no longer be effective later, so new evaluations need to be done over time. Also, she said, the dosage is critical to program outcomes, as is training, technical assistance, and performance measures to ensure quality and rigor.

More time and emphasis need to be spent on program selection, fit, and implementation, said Margolis. Many of the evidence-based programs identified by TPP have been around for a long time, and people are comfortable with them and have invested money in them. “But the decisions about which programs to implement need to be based on the needs of the community and what’s best for the community, and that takes time,” Margolis observed. Also, using evaluation results to inform program selection can be difficult, which sometimes means moving away from long-standing programs.

TPP is committed to using its evaluation data to continue to improve. It works with its trainees and trains them to collect data to make program decisions for continuous quality improvement. It has used evaluation results from the first cohort to inform the selection of programs for the second cohort. It spends time translating evidence for grantees and communities so

that they can understand and build that into their selection process. It also has identified areas in need of new and continued research.

As an example of how TPP has used evaluation results to inform the selection of programs by the second cohort of grantees, Margolis observed that it has not allowed new grantees to implement a program that has a negative evaluation. If three or more studies are unable to replicate findings from an original study, the program works with grantees to see if a different program might be a better fit in their communities. Evidence more than 20 years old is not considered. The program also is training its grantees and communities to understand what the evidence means, in part by translating the evidence to make it easier to understand. It has added more detailed information to implementation reports while also generating at-a-glance charts that show the differences right away in a single place. It has done interviews with developers to produce nonwritten forms of information. It also has worked with grantees to generate best practices, tips, and possible adaptation. Margolis noted, “We’re trying to help folks understand the differences between programs so that they can make decisions based on fit.”

With the first cohort, TPP focused on implementing a small number of evidence-based programs. It is now moving away from reliance on a single evidence-based program toward providing adolescents with multiple evidence-based programs. “Something in middle school, something in high school, something when you go to the clinic, something in the community.” Margolis said. “We’re talking about evidence-based programs in multiple settings and within a more holistic framework.” It has emphasized community mobilization, collective impact, engagement of youth and families, encouragement that environments are safe and supportive, trauma-informed services, and linkages to health care while keeping the focus on data, evaluation, dissemination, and sustainability, Margolis said.

UNLEASHING THE POWER OF PREVENTION THROUGH EVIDENCE-BASED PROGRAMS

The case also can be made that interventions based on principles rather than programs are not well enough understood to be widely disseminated. Gilbert Botvin, professor emeritus of health care policy and research at Cornell University’s Weill Medical College, made this point by observing that most early prevention efforts focused on trying to increase knowledge or change attitudes. But even where these efforts were able to do that, they tended to have little impact on target behaviors, such as tobacco, alcohol, and illicit drug use. In some cases, increasing knowledge or changing attitudes has increased drug use. “Untested ideas are not just ineffective,” Botvin observed, “they actually can make things worse.”

A new approach based on risk and protective factors and on a much greater understanding of the developmental course of problem behaviors has radically changed this situation. Interventions based on this new approach have progressed through pilot studies, small efficacy trials, effectiveness trials, and dissemination. The result is more than 60 programs demonstrated to be effective in preventing substance abuse, delinquency, violence, mental health problems such as depression and anxiety, and other behavioral health problems. Government agencies and professional organizations have identified and acknowledged evidence-based programs.² Furthermore, research has shown that prevention can save anywhere from $2 or $3 for every $1 invested to as much as $30 or $40 (Lee et al., 2015). Botvin said, “We’ve made a great deal of progress.”

As an example of an evidence-based program, Botvin briefly described the LifeSkills Training he has helped develop. Designed for middle school children, the program teaches a set of self-management and general social skills to decrease the internal motivations to engage in substance abuse and violence. It seeks to decrease vulnerability to influences from the media and from one’s peer network that promote substance abuse and violence and to promote resilience. Initially a 1-year program, it now consists of 15 class periods in the first year, 10 in the second year, and 5 in the third year. Interactive teaching methods get students involved and engaged in learning the content; training for teachers and other health professionals helps them implement these programs; and technical assistance enables them to overcome obstacles. More than 30 published studies have demonstrated the effectiveness of the program with diverse populations and different providers, with a benefit-cost ratio of 15 to 1, Botvin observed.

LifeSkills Training demonstrates many of the advantages of proven programs. They have detailed protocols and are carefully operationalized. They use well-tested methods, such as interactive teaching approaches. They contain user-friendly materials that make it easy for providers to implement the program. They can use existing dissemination structures while making available training and technical assistance. And tools exist for assessing implementation fidelity on a granular level, Botvin noted.

Botvin drew on his experience with this program to list some of the disadvantages of disseminating interventions based on principles. Currently, not enough understanding exists of the active ingredients or core components of programs, he said. “If we’re to move toward principles-based interventions, to a large extent, at this moment in time, we would be making educated guesses about what the active ingredients are,” he stated.

Second, data amassed over the years show that if programs are not implemented as designed, they are less effective. “The further they depart

___________________

² For more information, see http://www.blueprintsprograms.com/programs [May 2017].

from the design or the program as it was intended, the less effective they tend to be,” Botvin said. With a principles-based approach, the specification and operationalization of the principles are likely to be highly variable.

The dissemination of principles rather than programs also risks encouraging ineffective interventions. “The possibility is that we may, at least initially, give license to people in various settings to essentially go back to implementing their own programs or implementing programs that may be similar to the programs that have been tested and shown to be effective but really are not effective,” Botvin said. “And, by doing that, we could essentially undermine the progress that we’ve made in the field.”

Though it may be possible eventually to reach the point where principles-based interventions are effective, doing so today would be premature, Botvin said. “To unleash the power of prevention and capitalize on the past 35 years of research, it’s important at this point in time to stick to programs that have been tested and proven to be effective,” he concluded. “The emphasis now should be more on promoting the dissemination of programs and support systems that are likely to be effective and have been proven to be effective.”

DISCUSSION

Kernels versus Practices

The discussion of kernels and practices carried into the question-and-answer period. Biglan began by making a distinction between kernels and principles, defining a kernel as an instantiation of a principle. For example, a principle might be reinforcement of positive behaviors in children, while a kernel is a specific way to perform that reinforcement.

Patrick O’Carroll issued a caution: “We’re almost arguing about how to figure out which of the thousands of chemicals in kale is really good for you versus eating kale,” he said. Many programs, whether on teen pregnancy prevention, drug abuse prevention, teaching social skills, relationship development, or healthy sexual development, can catalyze each other, so that the whole is greater than the sum of the parts. “If we atomize this down to the kernel level to see what components should be in it, we may miss the catalytic sort of interactive effect of an environment that’s, broadly speaking, well informed and healthful in a variety of dimensions. That’s not to say we shouldn’t do this work, but I am curious how that larger picture would be integrated with the kind of analyses you’ve been describing,” he said.

David Hawkins, University of Washington and moderator for the panel, noted that in the Communities That Care prevention system, communities receive help to make their own decisions about what programs need to be

instituted to address prevalent risks or strengthen protective factors. At the same time, the program teaches social development strategies to create opportunities for activities that are developmentally appropriate for children. “We want to make sure they have the skills to do that, and we want to reinforce them in ways that reinforce their efforts,” Hawkins explained.

In places, programs and policies overlap to such an extent that the more appropriate focus is on functioning. “Let’s stop talking programs,” said Marc Atkins, University of Illinois at Chicago. “Let’s start talking functioning. Meaning, what are our outcomes? Where do we want to get to?” As August said, a more effective approach than programs in the school may be to have prevention delivery systems that are plugged into schools and provide screening and intervention. Such an approach would also resolve what Hawkins called the “big fight between people who are advocating policies and people who are advocating programs.” For instance, the implementation of evidence-based parenting programs in primary care settings underwritten by funding from Medicaid and private insurance would constitute a change in policies that would make multiple effective programs more widely available, Atkins suggested.

Biglan pointed to parallels in the tobacco control movement. “Most of the impact on smoking came through mass media and policies that changed as a result of mass media. And I can’t help but wonder, if we get better and better at communicating to parents the importance of patience and caring and compassion and all the things that are inside all of these programs, that we couldn’t have an effect simply by changing norms and public understanding,” he said.

Prevention and Health Care Reform

The panelists and workshop participants also explored the links between preventive programs in communities and ongoing reforms in health care. As Biglan, who is working with two coordinated care organizations in Oregon, pointed out, much of the money in health care is spent on people whose problems could have been prevented. Smoking, substance abuse, and depression all contribute to health problems, and all are targets of preventive programs. In addition, data being gathered under health care reform, both at the level of individuals and the community, could feed into the design and support of preventive programs.

Biglan also turned the conversation to a topic that became increasingly salient as the workshop progressed: the role of the social determinants of health. For example, stress in children can trigger inflammatory processes that lead to later health problems, he observed, so that the risk of adult heart diseases increases with more adverse childhood experiences. But with 20 to 40 percent of children in the United States living in pov-

erty, and with high rates of abuse and neglect, many children are being submitted to these stresses. Preventive programs will have a limited impact unless the United States reduces the proportion of children who are living in poverty, Biglan said. “These are huge stressors. We need public policies that change that. And we need to look at that larger social system. There is good research being done on policy that could guide us, but we also need to figure out how to evolve a capitalist system that’s built on the goal of ensuring the well-being of every person in society and getting past the view that if I selfishly pursue my own well-being economically, it will benefit everyone,” he said.

As Christopher Harris, Bright Star Church, pointed out, community members tend to think that intervention suggests “Let me fix you,” while prevention suggests “Let me make sure you never get broken.” What is really necessary, said Harris, is “real economic investment into communities that need these programs. . . . The greatest protective factor is intentional and significant community economic investment.”