Appendix D

Methodology for the Committee’s Data Analyses

The committee was tasked with conducting its own data analyses to address questions in its charge. The committee engaged a subcontractor, Westat, to carry out the data analyses. Per contractual requirements, the committee did not have access to the protected data provided by the Department of Defense (DoD) and relied on Westat’s analytic summaries for its deliberations and report. The Institutional Review Board of the National Academies of Sciences, Engineering, and Medicine approved the study.

DESCRIPTION OF DATA REQUEST

The data requested through the Defense Health Agency (DHA) from the Military Health System Data Repository included patient-level encounter data, claims data, and Comprehensive Autism Care Demonstration (ACD) records data from calendar years 2018 through 2023 for individuals aged 1 to 18 with autism. For the data request, autism was defined by an F84.0 International Classification of Diseases (ICD)-10 code between 2018 and 2023 in the patient record. Administrative data from claims and electronic health records (EHRs) included approximately 300 fields selected from seven DHA and DoD datasets.

The seven datasets requested included the (1) Comprehensive Ambulatory/Provider Encounter Record (CAPER), (2) TRICARE Encounter Data–Institutional (TED-I), (3) TRICARE Encounter Data–Non-Institutional (TED-NI), (4) Standard Inpatient Data Record (SIDR),

QUALITY CONTROL PROCESS

Westat closely followed committee definitions of cohort populations and relevant covariates. Researchers collaborated with the committee to iterate over definitions and criteria for data harmonization and analysis. In addition, Westat researchers and programmers iterated over implemented logic and ensured matching results and populations. Epidemiologists at Westat reviewed output and code from programmers to ensure specifications were implemented.

SAS PROGRAMS

Several SAS scripts were created for this study and can be grouped by purpose.

1. Identification of patients with ABA claim—ABA assessment and treatment codes were extracted via Current Procedural Terminology (CPT) procedure codes and saved as SAS datasets
   1. Sorted into continuous and intermittent cohorts
2. Identification of non-participants
3. Identification of relevant ICD-10 diagnosis codes and CPT treatment codes—ICD lookup tables were manually created to be used by the scripts. Relevant ICD-10 data were extracted and saved as SAS dataset
   1. Lookup table
4. Cleaning DEERS data
5. Cleaning non-DEERS (TED-NI, GENESIS, CAPER) data
6. Creating Table 1 Analysis Dataset
   1. Creation of cohort-level analytic file—append demographic data to cohort
   2. Creation of analytic file—append data for all cohorts
7. Creating Table 2 Analysis Dataset
   1. Creation of cohort-level analytic file—append ICD-10 diagnosis data to cohort
   2. Creation of analytic file—append data for all cohorts
8. Creating Table 3 Analysis Dataset
   1. Calculated ABA intensity
   2. Creation of cohort-level analytic file—append ICD-10 diagnosis data to cohort
   3. Creation of analytic file—append data for all cohorts
9. Creating Table 4 Analysis Dataset
   1. Creation of cohort-level analytic file—append ICD-10 diagnosis data to cohort
   2. Creation of analytic file—append data for all cohorts

ANALYTIC APPROACH

Given limitations with the data and time available to conduct the analyses, the committee undertook analyses aimed at understanding similarities and differences between autistic individuals who participated in the ACD and those who did not. Participants in the ACD program were further divided into continuous (greater than six months of continuous ABA treatment) and intermittent participants (all others with receipt of some ABA treatment). Analyses compared demographics, clinical characteristics, as well as the amount of ABA received across groups.

Demographics by ACD Participation

Demographic variables were obtained through several data sources such as DEERS, TED-NI, CAPER, and GENESIS datasets. Linking these data resulted in variable information by beneficiary. In addition, recorded demographics changed over time within the same datasets. For DEERS data, if demographic data were not able to be merged by date, the closest record to the relevant date was kept. To mitigate these differences, mismatching demographics were set to missing, unknown, or other values.

Descriptive analysis compared frequencies and means of demographic characteristics among continuous participants, intermittent participants, and those not participating in the ACD. These analyses are further broken out by age group. Since ABA is only authorized through the ACD program in TRICARE, beneficiaries with at least one ABA claim were considered ACD participants.

• Continuous participants were defined as beneficiaries with an F84.0 ICD-10 code in their medical records and at least one ABA claim per month for at least six consecutive months between January 1, 2018, and December 31, 2023.
• Intermittent participants were defined as beneficiaries with an F84.0 ICD-10 code in their medical records and at least one ABA claim between January 1, 2018, and December 31, 2023. The intermittent participants infrequently reached six consecutive months of ABA therapy.
• Non-participants were defined as beneficiaries with an F84.0 ICD-10 code but with no ABA-related claims in their medical history during this this time frame.

Westat created descriptive tables comparing consistent, intermittent, and non-participants for ACD by the following demographic, military, and autistic characteristics. See Table D-1 for an example.

For demographic analyses, characteristics were defined by reported demographics in DEERS and matched to the nearest record in DEERS to

TABLE D-1 Sample Table for Demographic Characteristics Examined for Ages 0–2

an individual’s first autism ICD-10 code (F84.0). This approach was used for all variables and analyses unless otherwise specified.

• Age
  • Determined by first F84 diagnosis in clinical datasets between 2018 and 2023. Birthdate obtained from DEERS. Age is as of January 1 for any given year.
• Sex
  • Male
  • Female
  • Missing
• Ethnicity (Note: Provided by DHA)
  • Hispanic
  • Non-Hispanic
  • Missing
• Race (Note: Provided by DHA)
  • White
  • Black
  • Asian
  • Other
• Sponsor rank
  • Officer (Junior officer, senior officer, warrant officer)
    • Enlisted (junior enlisted, senior enlisted)
  • Other (cadet, missing)

• Region
  • Alaska
  • East
  • West
  • Outside the Continental United States
  • Missing
• Sponsor branch
  • Army
  • Air Force
  • Coast Guard
  • Marines
  • Navy
  • Space Force
  • Missing
• Sponsor component
  • Active Duty
  • Guard/Reserve
  • Retired
  • Missing
• Urbanicity (grouped into the percentage of population within five-digit ZIP codes rolled up to a three-digit ZIP code)
  • 0–50% Urban
  • 50–75% Urban
  • 75–90% Urban
  • 90–99% Urban
  • 100% Urban
• Moved (permanent change of station [PCS])
  • Yes
  • No
  • Defined as a change in ZIP code for the patient during the first six months for users of continuous or intermittent ACD, or six months after F84.0 diagnosis for non-participants
• Baseline PDDBI used as a marker of autism severity

Clinical Outcomes

Diagnostic specific covariates were derived using ICD-10 codes from TED-I, CAPER, SIDR, GENESIS, and TED-NI datasets.

Clinical analysis compared frequencies and means of clinical diagnosis or treatments between continuous participants, intermittent participants, and those not participating in the ACD. Treatments frequently received (e.g., speech, occupational therapy [OT]/physical therapy [PT]) and co-occurring conditions (e.g., intellectual disability) with autism were identified by the

TABLE D-2 Clinical Characteristic Fields

NOTE: ADHD = attention-deficit/hyperactivity disorder; ENT = ear, nose, and throat; ICD = International Classification of Diseases; OCD = obsessive-compulsive disorder.

TABLE D-3 Clinical Characteristics among Ages 0–2

NOTES: ADHD = attention-deficit/hyperactivity disorder; ENT = ear, nose, and throat; OCD = obsessive-compusive disorder; OT = occupational therapy; PT = physical therapy.

TABLE D-4 Clinical Characteristics among Age 3

NOTES: ADHD = attention-deficit/hyperactivity disorder; ENT = ear, nose, and throat; OCD = obsessive-compusive disorder; OT = occupational therapy; PT = physical therapy.

TABLE D-5 Clinical Characteristics among Age 4

NOTES: ADHD = attention-deficit/hyperactivity disorder; ENT = ear, nose, and throat; OCD = obsessive-compusive disorder; OT = occupational therapy; PT = physical therapy.

TABLE D-6 Clinical Characteristics among Age 5

NOTES: ADHD = attention-deficit/hyperactivity disorder; ENT = ear, nose, and throat; OCD = obsessive-compusive disorder; OT = occupational therapy; PT = physical therapy.

TABLE D-7 Clinical Characteristics among Ages 6–11

NOTES: ADHD = attention-deficit/hyperactivity disorder; ENT = ear, nose, and throat; OCD = obsessive-compusive disorder; OT = occupational therapy; PT = physical therapy.

TABLE D-8 Clinical Characteristics among Ages 12–18

NOTES: ADHD = attention-deficit/hyperactivity disorder; ENT = ear, nose, and throat; OCD = obsessive-compusive disorder; OT = occupational therapy; PT = physical therapy.

included overall and communication scores from the SRS. The committee determined that domain specific subscores would more appropriately capture behaviors that ABA could be used to address over the total score of the PDDBI. Although these subscores are important, they were not necessarily used by clinicians to measure behaviors unique to the individual.

The goals of ABA, and the corresponding assessment measures, are not captured in these data and substantial misclassification likely exists.

Descriptive analyses were performed to understand the utility of assessment scores: determined the number of assessments by year and compiled the number of beneficiaries with valid assessments (by number of assessments completed). In doing this, it was revealed that over half of the assessments in the database did not contain a date of assessment but rather included only the year/quarter when the data were ingested into the DHA database, which does not correspond to when the assessment was performed. Westat also identified issues with multiple scores in the same time period or even the same date, but this problem was much smaller than the missing valid dates. Assessments with missing scores were removed when reporting by subscores.

Baseline score was defined as the first recorded assessment for any beneficiary. Preliminary analysis examined the average assessment score and standard deviation by demographic characteristics outlined previously. In this case, demographics (e.g., age) were matched to the nearest date of the assessment, not the first F84.0 code. In addition, Westat also produced a table comparing the scores for first and second assessments for individuals with at least two assessment scores. To determine the impact of the missing dates, Westat performed this analysis among the total beneficiary population with two scores and valid assessment dates and then again among those with two valid dates of assessments.

Preliminary Outcome Modeling

Associations between ABA intensity and changing score from the PDDBI REXSCA (receptive/expressive social communication abilities dimension) were examined in a sample of 311 autistic individuals using a paired t-test and linear regression. The initial model examined predictors of the baseline PDDBI REXSCA score using linear regression that included age, sex, and ABA intensity groups. This was followed with a paired t-test, testing mean differences between the first and second assessment. Thirdly a linear regression was used with the second assessment as the dependent variable with age, sex, ABA intensity, and the initial assessment score as independent variables. Older age groups and males had a higher first assessment mean REXSCA score. The paired t-test did not detect a difference between mean first and second assessment scores. The older age group (≥10)

The committee also performed a preliminary analysis of 1,495 ACD participants who had received ABA and had been assessed twice by Vineland-3. At baseline, the average Vineland-3 score was 70. The amount of ABA later received was positively associated with the baseline score. Westat also constructed a model to examine changes in Vineland scores over time among these participants controlling for amount of ABA, sex, and age group. Small overall improvements in the sample were observed, with association more pronounced among three-to-six-year-old children. And note that improvement did not vary with amount of ABA (i.e., observe similar levels of improvement across the groups defined by amount of ABA [i.e., 0–5, 5–10, 10–20, and 20 or more hours per week]).

Raw R output from the Vineland ABC model (N = 1,495)

Multivariable linear regression of intensity, age, and sex on the first Vineland ABC score.

Multivariable linear regression of intensity, age, and sex on the second Vineland ABC score while controlling for the mean centered first Vineland ABC score.

DATA LIMITATIONS

EHR systems do not collect data with research-based objectives in mind and can be incontinuous across data fields that would inform specific research questions. Variables may have high levels of missingness. For example, the DHA has communicated that large proportions of race/ethnicity data are missing. There are additional challenges stemming from omissions common in administrative data beyond inconsistencies and missingness. Claims-based data will not capture information beyond what is needed for a billable claim, despite its relevancy for research questions.

The ability to capture breaks in care, and their cause, can be challenging in claims-based data sources. Breaks in care can be uniquely detrimental to individuals with autism as autistic individuals benefit from consistency and routine. This is exacerbated in this study because military families experience frequent breaks in care due to PCS relocations, which happen every two to three years. It is difficult to ascertain when care was stopped due to a factor that may be captured by administrative data, such as referral exhaustion, or a reason not included such as choosing to not participate in ABA. ABA is intended to be a continuous treatment with the same provider or set of providers so changes to ABA providers, and delays associated with those changes, could impact outcomes.

Claims records will have important historical components omitted as well. The time that ICD codes were recorded may not, in and of itself, identify when the condition was originally recognized. For example, an individual could join TRICARE after an autism diagnosis was recognized in a different healthcare system. The age of an individual at diagnosis has been suggested by the committee as a marker of autism severity. And in turn autism severity could be a strong predictor of service utilization.

Claims-based data omissions can lead to biases if comparisons are made between populations with important unmeasured factors. This study is not a randomized control study, making careful development of an