4

A Program Evaluation Overview for Support of Pilots and Flight Attendants with Substance Use Disorders

Pilots and flight attendants operate in a stressful environment that is shaped by public policies and expectations for accountability. This chapter illuminates the larger context in which pilots and flight attendants work, with the goal of describing how evaluation could be employed to better understand whether current supports are adequate for addressing substance use disorders and ensuring their well-being. For this purpose, the chapter first provides a framework that outlines the multiple influences on the work of pilots and flight attendants and includes public policies, airline policies and cultures, workplace stressors, and individual factors. Next, it draws upon the findings laid out in Chapter 3 on the key elements of evidence-based practices for identification and treatment of substance use disorders, and then reviews best practices in program evaluation, to propose key elements to include in evaluations of substance use disorder programs. Finally, evaluation tools used by the Human Intervention Motivational Study (HIMS) and the Flight Attendant Drug and Alcohol Program (FADAP) are discussed in light of how well these align with best practices for program monitoring and assessment. This is followed by recommendations for ways the Federal Aviation Administration (FAA) can improve oversight of substance use disorder programming.

Many stressors can affect the performance of both pilots and flight attendants, but the legally required testing and subsequent severe penalties for alcohol and drug use placed on pilots in particular impose even more stress on pilots, some of whom may be struggling with personal issues that result in their coping through substance use.

FACTORS THAT FACILITATE OR IMPEDE PILOTS AND FLIGHT ATTENDANTS FROM SEEKING TREATMENT WHEN NEEDED

This section reviews four types of factors that may facilitate or impede pilots’ and flight attendants’ willingness to seek treatment when needed. As shown in Figures 4-1 and 4-2, the four types include input factors, external factors, individual factors, and organizational factors that lead to desired outcomes of capable pilots, flight attendants, and airline safety.

Inputs

Input factors are those that may influence an employee’s decision to pursue substance use disorder treatment. These include pilot/flight attendant characteristics, public accountability, and policies at various levels that may dictate or influence behavior. Pilot-related characteristics may impact whether and how they seek treatment when they have substance use disorders. DeHoff and Cusick (2018) also point out that the distinctive nature of the occupation of pilots may also impact their willingness to seek help when they have depression, anxiety, and other mental illness. Pilots are more prone to hazardous attitudes such as invulnerability and macho (e.g., “It won’t happen to me” and “I can do it”; DeHoff & Cusick, 2018), which is likely to prevent them from seeking help when needed.

Public accountability is another significant factor, one that impacts pilots’ and flight attendants’ tendency to seek medical treatment. The physical and mental health of the pilots and flight attendants is important to the safety of flight, which is critical to the airline, the passengers, and the public. Indeed, flight accidents are usually associated with serious injuries and fatalities, which cause severe loss to the public and cumulative decreases in enplanement for the airlines (Insua et al., 2019 Squalli & Saad, 2006). Therefore, pilots, flight attendants, and airlines share the responsibility to ensure the safety of their flights. The policies of the FAA, unions, and airlines may also influence pilots’ and flight attendants’ willingness to seek treatment. For example, flight crews may hesitate to seek help due to a concern about the potential impacts on their medical certification process. The Guide for Aviation Medical Examiners requires that an employee reporting any mental condition “requires investigation through supplemental history taking,” that “dispositions will vary according to the details obtained,” (FAA, 2023b), and that this finding be reported to the FAA (DeHoff & Cusick, 2018).¹ Consequently, flight crews’ perceived risks of interrupting their careers may be a hurdle for them to seek needed assistance. In addition,

___________________

¹ After a prepublication version of the report was provided to the FAA, this section was edited to reflect 2023 FAA Guidance.

airlines’ policies may also play a similar role in influencing pilots’ and flight attendants’ attitudes about seeking treatments. In a study that sampled 221 flight department leaders, 68 percent of respondents indicated that their flight operation had a written substance use policy, but only 20 percent indicated that it had policies on rehabilitation and only 16 percent that it had job retention/recovery policies (Vardiman, 2008). As a result, the flight crews’ fear of disrupting their careers may lead to their unwillingness to seek medical treatment when they are involved in substance abuse disorders.

External Factors

Peer support can play a key role in facilitating pilots and flight attendants in obtaining recovery. Alcoholics Anonymous (AA), for example, provides those who have substance use disorders an opportunity to learn about addiction at no cost and from others in a similar situation. It has a low threshold of inclusion, encourages expressions of one’s own personal history and challenges, and helps people build intimate social networks (Westermeyer, 2014). AA and other mutual help groups can be especially helpful when involved individuals share similar experiences and characteristics (e.g., Birds of a Feather, Wings of Recovery or similar cultures; Westermeyer, 2014). These groups also help people recognize that while their profession or socioeconomic status may differ from others’ in these groups, they have the same disease and have more in common than they initially perceived. Both FADAP and HIMS include peer support as a foundation of their program.

In addition to the social-emotional support from peers, professional support from addiction experts and treatment programs are also important facilitators of flight crews’ health and well-being. When pilots or flight attendants have substance use disorders, appropriate medical services and treatments are critical for their recovery (Horton et al., 2010, 2011; Shorey et al., 2014). HIMS and FADAP are designed to connect employees with such treatment.

While many external facilitators can help flight crew members obtain treatment when needed, other external factors may stop them from receiving these treatments and make their situation more stressful. For example, one important factor is related to their family. While supportive families can be key to encouraging help seeking, pilots and flight attendants are prone to having strained family dynamics. Difficulty finding time they can spend with their families during peak work times and missing important family events due to their work responsibilities (Cullen et al., 2021) causes strain. For this reason, their spouses may also suffer due to the inability of pilots and flight attendants to take up their share of domestic or parental roles (Cullen et al., 2021). Additionally, due to the limited usage of cell

phones on flights, pilots and flight attendants may also find it hard to participate in school-related activities for their children (Butcher, 2002; Foster & Ren, 2015; Ren & Foster, 2011). All these pressures related to family issues can add to the stress in pilots’ and flight attendants’ lives and may even intensify their reliance on alcohol or drugs.

Another, more recent, adverse factor that has emerged is COVID-19. Since the outbreak of the global pandemic in early 2020, the economy has inevitably suffered greatly, especially in the travel industry. Some areas were locked down, and others set strict policies about passengers’ health status. Consequently, the airlines have experienced significant decreases in revenues. This external shock has led to uncertainty and insecurity for pilots and flight attendants regarding their future careers and their medical certifications (Suthatorn & Charoensukmongkol, 2022; Troyer & Bidaisee, 2022; Xiao et al., 2022), which makes their situation worse, especially for those with substance use disorders.

The projected industrywide pilot shortage may also aggravate pilots’ and flight attendants’ occupational stressors and prevent them from accessing effective treatment. Klapper and Ruff-Stahl (2019) estimate that regional airlines would face a deficit of about 5,000 to 8,000 pilots by 2023. This may be translated to increased workloads and reduced break time, thereby creating stressors and reduced possibility of getting timely treatments.

Individual Factors

A range of individual factors can also facilitate or impede pilots’ and flight attendants’ treatment-seeking when they are in need. Resiliency represents an individual’s strengths and means of coping with stressors and adversity (Richardson, 2002). Resilient individuals are more likely to successfully handle stressors and deal with disruptions, and thus help themselves heal from the harmful impacts of substance use with the right treatment. Self-efficacy is the belief in one’s abilities to produce designated levels of performance and exercise influence over events that affect their lives (Bandura, 1982). Those with higher levels of self-efficacy may believe that they can change their own situations and may thus spend efforts to seek treatments for their own recovery. While research has also shown that self-efficacy is an important predictor of post-treatment abstinence, this can also be a barrier to treatment success for professionals, who can be too self-reliant, limiting their willingness to ask for or otherwise seek help (Kadden & Litt, 2011).

Healthy eating and exercise can also help pilots and flight attendants in this situation, just as they have been found to benefit physical and mental health in general. Indeed, healthy eating and exercise can serve as preventive and therapeutic interventions to prevent and treat mental illness and

substance use disorders (Zschucke et al., 2012). In addition to the factors related to themselves, factors related to their significant others may also play a role. When pilots and flight attendants receive family support (e.g., financial support, direct care), they are more likely to take action and seek treatments when needed. In fact, family support in the form of economic assistance and direct care has been linked to substance abuse recovery and substance use reduction (Clark, 2001).

Despite these positive factors, some other individual factors may make pilots and flight attendants more prone to substance use disorders or impede their help-seeking (as described in-depth in Chapter 3). Genetic predisposition is a critical individual factor that helps explain the between-person differences in addiction tendency. Research has found that addiction runs in families (Hall et al., 2015). More direct evidence on human DNA has also provided preliminary support for the potential linkage between genes and addiction (Ball, 2008). Shame is another individual factor that may influence pilots’ and flight attendants’ tendency to receive treatment. Those with substance use disorders commonly have shame and blame themselves for such problems (O’Connor et al., 1994), which may worsen their situations and even stop them from seeking treatment, adding to internalized stigma related to mental health, as discussed more thoroughly in Chapter 3.

Organizational Factors

Several nonregulatory organizational factors may also impact pilots’ and flight attendants’ willingness to seek treatment when in need. The level of airline support is one of the most important factors, because pilots’ and flight attendants’ perceived uncertainty about how treatment or medical history impacts their careers has been identified as one of the most important considerations when they have alcohol or other drug-related problems (DeHoff & Cusick, 2018; Vardiman, 2008).

Airline Financial Support

Inadequate financial support from airlines for treatment presents an especially important factor. For example, pilots are required to undergo neuropsychological and psychiatric assessments to assess fitness for duty, as well as assessments of the quality and extent of recovery in preparation for application for Special Issuance authorization. However, these required evaluations are typically not reimbursable by insurance carriers, because the evaluations are geared toward regulatory requirements and not clinical diagnostics. These evaluations are often paid for by larger airlines in accordance with collective bargaining agreements. On the other hand, pilots for smaller carriers and regional airlines may have less financial support for

these assessments, which can be extremely costly as discussed in Chapter 2 (see cost estimates by treatment modality).

Similarly, airlines vary greatly in the amount of financial support they provide flight attendants who seek treatment. Unlike pilots, for whom the treatment is typically fully covered, flight attendants may face extremely high out-of-pocket costs for treatment, even up to a large proportion of their annual income. Lower out-of-pocket costs would help both flight attendants and pilots receive appropriate treatments, as financial considerations have an unignorable impact on patients’ willingness to receive medical services (Clark, 2001). Flight attendants in general, and pilots at smaller airlines, are clearly at more risk of not seeking treatment due to the heavy financial burden they would face. Although we do not know much about the practices in industries with other transportation modes (e.g., trucking, train, shipping), the recommendation for the employer to cover substance use disorder treatment seems to equally apply to other safety-sensitive positions.

Organizational and Psychological Safety Climate

The climates of organizational safety and psychological safety are also important factors affecting the likelihood that pilots and flight attendants will seek treatment for substance use disorder. Within a psychologically safe climate, pilots and flight attendants are less likely to have perceived uncertainty and job insecurity, which may deter them from seeking treatment (Edmondson, 1999). The Department of Labor has resources and guidance for any organization willing to establish a Recovery-Ready Workplace,² and several industries with particularly high-stress and safety-sensitive environments have called for an organizational climate of safety and wellness; these industries include (but are not limited to) physicians and nurses (Baldisseri, 2007; DuPont & Skipper, 2012; Shaw et al., 2004), trucking (Arboleda et al., 2003), and law enforcement (Taylor et al., 2022).

The U.S. Department of Transportation (DOT) has defined safety culture as “The shared values, actions, and behaviors that demonstrate a commitment to safety over competing goals and demands” (Morrow & Coplen, 2017, p. 2). DOT further details the 10 most critical elements of a strong safety culture as these:

1. leadership is clearly committed to safety;
2. open and effective communication exists across the organization;
3. employees feel personally responsible for safety;
4. the organization practices continuous learning;
5. the work environment is safety conscious;

___________________

² https://www.dol.gov/agencies/eta/RRW-hub/Recovery-ready-workplace

6. reporting systems are clearly defined and not used to punish employees;
7. decisions demonstrate that safety is prioritized over competing demands;
8. employees and the organization work to foster mutual trust;
9. the organization responds to safety concerns fairly and consistently; and
10. safety efforts are supported by training and resources.

Similarly, Box 4-1 describes five important aspects of a safety culture.

Union Support

Union support may also affect whether or not pilots and flight attendants seek treatment, because unions may contribute to the improvement of relevant organizational policies for flight crew benefits, such as explicit policies about rehabilitation and job retention. Further, professional programs such as HIMS and FADAP can also help pilots and flight attendants when they are in need. However, while HIMS and FADAP can be helpful, it is likely that some pilots and flight attendants are unaware of the existence of these supporting programs. In fact, anecdotal input received by this committee during the committee meetings suggests that the visibility of both HIMS and FADAP can be bolstered.

It is worth highlighting key aspects of the work scheduling and environment that also may discourage pilots and flight attendants from seeking treatment. Long work hours may decrease their possibility of seeking (and getting) treatment because work occupies too much of their time, rendering them without the temporal and psychological resources to obtain medical services (Andresen et al., 2007). A “meet-at-the-bar” culture may also make it hard for the flight crew members to avoid alcohol or may even normalize their addiction so they stop seeking treatment (Liu et al., 2015). As noted above and discussed in Chapter 3, a punitive culture or the risk of losing employment may also be harmful to pilots’ and flight attendants’ health and their willingness to receive treatment (DeHoff & Cusick, 2018). In addition, stigma is also a major factor that prevents the flight crew from treatment and confounds their recovery (Corrigan et al., 2017). Because substance misuse is socially discrediting, those in need may mentally deny the existence of such problems or distance themselves from treatment to avoid being regarded as problematic (Corrigan et al., 2017).

KEY ELEMENTS OF GOOD SUBSTANCE USE DISORDER TREATMENT FOR PILOTS AND FLIGHT ATTENDANTS

HIMS and FADAP play important intermediary roles in helping pilots and flight attendants needing support to deal with substance use disorders find and complete appropriate treatment and reenter their work roles successfully. Not all pilots and flight attendants work with HIMS and FADAP to secure support, but the focus of this study is on how HIMS and FADAP can best serve those who do use their services. Both HIMS and FADAP have criteria to judge the providers to whom they refer pilots and flight attendants.³ Drawing upon the most up-to-date data to emphasize the best practices for substance use disorder treatment (see Chapter 3 for more details), this section outlines highlights and key takeaways.

As a starting point, a key question to ask is: What will success look like for the individuals and the other stakeholders, including unions, the airlines, and the flying public? (See Figure 4-3.) Success, measured by outcomes, may differ across types of stakeholders. “For [alcohol use disorders], health outcomes can be expanded to include four areas: sustained reductions in [alcohol use disorder] use, improvements in personal health, sustained improvements in functioning (e.g., employment), and sustained

___________________

³ For FADAP, this process includes conducting interviews both with the participating flight attendant, their peer mentor, and the treating provider and could include review of the flight attendant’s treatment file. Request for corrective action by the treatment program or suspension of referrals until corrective action takes place may be the outcome of the quality assurance process.

reductions in threats to public health and safety” (Garnick et al., 2006, cited in McLellan et al., 2005). FAA and airline management, for example, may place priority on workplace outcome measures invoking safety in flight operations such as long-term sobriety (abstinence) and rate of return to safety-sensitive duties while program outcomes may focus on treatment effectiveness (rate of relapse) and length of time to enter into treatment from time of identification/referral. From the perspective of individual pilots and flight attendants, success may be spending as little time away from the job as possible. But success in terms of obtaining sobriety and sufficient recovery may take much longer than the individual pilots and flight attendants prefer. Chapter 5 of this report presents statistics on program outcomes, in particular in Tables 5.3 through 5.5. The optimal combination of treatment and recovery support will vary based on the individual needs of a given pilot or flight attendant. There is no standardized treatment plan that will meet all pilot and flight attendant needs, but recognizing that there is not one perfect combination, nor one standard intervention timeline, is a first step.

Continuing care presents an especially important and essential support for pilots and flight attendants with substance use disorders (as described in depth in Chapter 3). It is essential that the recommended continuing care support services be individualized and person-centered, as stated by McKay (2009, p. 12):

Current research does provide guidance on the elements of substance use disorder treatment that have been shown to be effective. A key consideration is what is the most appropriate level of care for entry to treatment. While residential treatment is considered optimal by HIMS and FADAP, as distractions due to family and other demands are reduced, it is often perceived as especially burdensome and overly costly, particularly depending on insurance coverage and financial resources. Many essential elements of good programming for substance abuse disorder treatment were described, based on the available evidence, in Chapter 3. Qualified treatment programs will at least:

• be patient-centered and individualized;
• be guided by American Society of Addiction Medicine standards for assessment and use the biopsychosocial-spiritual model;
• be staffed with professionals holding expertise and advanced clinical credentials;
• offer evidence-based interventions for treating substance use disorders (e.g., appropriately selected and monitored medications, Motivational Interviewing, mindfulness strategies, cognitive behavioral therapy, dialectical behavior therapy, contingency management, and peer support); and
• be trauma-focused and address co-occurring physical and mental health conditions.

EMPLOYING EVALUATION TOOLS

Program evaluation is the application of systematic methods to address questions about program operations and results, and it includes ongoing monitoring of a program as well as one-shot studies of program processes or program impact. Evaluative thinking and use of data collection tools present valuable opportunities for enhancing knowledge about the underlying logic of programs and the program activities under way, as well as about the results of programs (Newcomer et al., 2015). The field of program evaluation offers processes and tools useful for obtaining valid, reliable, and credible data to address a variety of questions about the performance of any program.

A frequently used distinction in evaluation work is between formative and summative evaluation. Formative evaluation uses evaluation methods to improve the way a program is delivered, while summative evaluation entails systematically measuring program outcomes and results during ongoing operations or after program completion. Most evaluation work will examine program implementation to some extent, if only to ensure that the assessment of outcomes or impacts can be logically linked to program activities.

Credible evaluation work requires clear, valid measures that are collected in a reliable, consistent fashion. Evaluation work must begin with clarifying what will constitute credible measures and establishing strong procedures to ensure that both quantitative measurement and qualitative measurement are rigorous. The relevance, legitimacy, and clarity of the data collected on both program implementation and outcomes matter to program stakeholders and the public.

So, how might HIMS and FADAP employ evaluation effectively? Both formative and summative evaluation approaches are needed, as is the collection of valid and reliable data. In particular, data should be collected for the target population (i.e., employees with safety sensitive duties) to determine the effectiveness of prevention efforts as well as the prevalence of substance use disorder (both treated and untreated) in the workplace, in addition to data from all participants (rather than a sample of them) referred to the programs and following treatment completion (see Box 4-2). The research undertaken as part of the committee’s work suggests that more careful, complete, and transparent data collection is needed.

In addition to monitoring how well those who complete treatment fare, there are a number of other best practices that both HIMS and FADAP should consider. The committee outlines a set of questions that both programs should address on an ongoing basis to ensure that through their actions they are contributing to the well-being of pilots and flight attendants (see Table 4-1).

TABLE 4-1 Adherence to Recommended Practices—Considerations for HIMS and FADAP