7

Conclusions and Recommendations

ROLE OF PSYCHOLOGICAL TESTING IN SOCIAL SECURITY ADMINISTRATION DISABILITY PROGRAMS

The committee reached a number of general conclusions pertaining to the role of standardized psychological testing in the U.S. Social Security Administration (SSA) disability programs:

• The two largest impairment categories for Supplemental Security Income (SSI) (adults and children) and Social Security Disability Insurance (SSDI) are mental disorders (excluding intellectual disabilities) and musculoskeletal and connective tissue disorders. Within these two categories, a significant fraction of the applicants have conditions, including affective mood disorders and disorders of the back, for which the presence and severity of impairment and associated functional limitations are based largely on applicant self-report.
• SSA disability determinations are based on the medical and all relevant evidence in an applicant’s case record. Physical or mental impairments must be established by objective medical evidence consisting of medical signs and laboratory findings, which may include psychological and other standardized test results. SSA establishes the presence of a medically determinable impairment in individuals with mental disorders other than intellectual disability through the use of standard diagnostic criteria, which include symptoms and signs. Evidence for claims based on mental impairment, as well as for claims for conditions in which the somatic symptoms

• are disproportionate to physical findings (e.g., somatoform disorder, multisystem illness, and chronic pain), relies less on standard laboratory tests than for some other categories of impairment. The validity of the self-reported symptoms and/or impairment severity may be called into question due to the absence of objective medical evidence or biomarkers that could explain or substantiate the applicant’s self-report of distress and disability.

• In some cases, SSA disability examiners must evaluate the credibility of statements by individuals about the intensity and persistence of their symptoms and the effect on the individual’s ability to function and perform work-related activities. When a disability claim is based primarily on an applicant’s self-report of symptoms and self-reported statements about their intensity, persistence, and limiting effects, SSA relies on an assessment of the consistency of the self-report with all of the evidence in the claimant’s medical evidence record.
• There currently is great variability in allowance rates for both SSI and SSDI among states that is not fully accounted for by differences in the populations of applicants. In addition, there is great variability in the appeal rulings among administrative law judges within and across states.
• Psychological consultative examinations often consist of nonstandardized diagnostic interviews and a mental status exam, with little or no standardized psychological testing. Because clinicians generally are not as good at interpreting clinical and standardized test data as are established actuarial methods, reliance on established actuarial methods (when available) to interpret the data will improve the accuracy of diagnostic evaluations.
• Each Disability Determination Services (DDS) agency, within the confines of SSA policy, issues its own rules regarding the tests that may be purchased as part of a consultative examination. Aside from the use of intelligence tests as described in the listings for intellectual disability and certain neurological impairments, SSA does not require or specify the purchase of any type of (or individual) psychological test. SSA provides general guidance that good psychological tests are valid and reliable and have appropriate normative data. For this reason, there is variation among states about when and which standardized psychological tests can be purchased, with the exception of performance validity tests (PVTs) and symptom validity tests (SVTs), which are precluded from purchase by SSA except in rare cases such as a court order.
• The results of standardized cognitive tests and non-cognitive psychological tests that are appropriately administered, interpreted,

• and validated can provide objective evidence to help identify and document the presence and severity of medically determinable mental impairments at Step 2 of SSA’s disability determination process. In addition, standardized cognitive test results can provide objective evidence to help identify and assess the severity of work-related cognitive functional impairment relevant to disability evaluations at the listing level (Step 3) and to mental residual functional capacity (Steps 4 and 5).

• Current data on the prevalence of inconsistent reporting of symptoms or performing below one’s capability on cognitive tests are very imprecise. In the context of SSA disability applicants, neither scenario rules out disability, but both suggest the need for additional assessment of the alleged impairment with the goal of making an accurate determination of disability.
• SVTs and PVTs provide information about the validity of standardized non-cognitive and cognitive test results when administered as part of the test or test battery and are an important addition to the medical evidence of record for specific groups of applicants. Validity tests do not provide information about whether or not the individual is, in fact, disabled.
• Because SVTs and PVTs are used to help assess the validity of an individual’s standardized non-cognitive and/or cognitive psychological test results respectively, it is important that SVTs and PVTs only be administered in the context of a larger test battery and only be used to interpret information from that battery.
• Current SSA policy precludes the purchase of SVTs and PVTs to help inform determinations about the credibility of an individual’s statements or about possible malingering. Specific tests outlined as examples in this policy include not only stand-alone PVTs and SVTs (e.g., Test of Memory Malingering, Validity Indicator Profile, Structured Interview of Reported Symptoms), but also psychological self-report measures that contain symptom validity scales (e.g., Minnesota Multiphasic Personality Inventory-2, Millon Clinical Multiaxial Inventory) among other scales of psychological functioning. This policy is inconsistent with the practice of other disability benefit programs, such as the Veterans Benefits Administration, private disability insurers, and some international disability programs.
• Although there currently are no data on the rates of false positives and false negatives in SSA disability determinations, systematic use of standardized psychological testing for a broader set of physical and mental impairments than is current practice is expected to improve the accuracy and consistency of disability determinations

STANDARDIZED NON-COGNITIVE PSYCHOLOGICAL MEASURES AND SYMPTOM VALIDITY TESTS

The following conclusions and recommendation pertain specifically to the use of standardized non-cognitive psychological measures and associated SVTs in SSA disability determinations:

• The use of standardized non-cognitive psychological measures is essential to the determination of all cases in which an applicant’s allegation of non-cognitive functional impairment meets each of three requirements:
  1. The applicant alleges a mental disorder (i.e., schizophrenic, paranoid, and other psychotic disorders; affective disorders; anxiety-related disorders; and personality disorders) unaccompanied by cognitive complaints or a disorder with somatic symptoms that are disproportionate to demonstrable medical morbidity (i.e., somatoform disorders, multisystem illnesses, and chronic idiopathic pain conditions).
  2. The presence and severity of impairment and associated functional limitations are based largely on applicant self-report.
  3. Objective medical evidence or longitudinal medical records sufficient to make a disability determination do not accompany the claim.
• In certain instances, cognitive concerns may accompany the applicant’s allegations, in which case cognitive testing, as discussed below, may be more appropriate. The committee also recognizes that there are a few chronic conditions (e.g., schizophrenia, chronic idiopathic pain, multisystem illnesses) that may generate potentially disabling, non-cognitive functional impairments but may not be accompanied by objective medical evidence. In such cases, the evidence provided by longitudinal medical records may be sufficient to substantiate the allegation.
• Assessment of symptom validity, including the use of SVTs, analysis of internal data consistency, and other corroborative evidence, helps the evaluator to interpret the accuracy of an individual’s self-report of behavior, experiences, or symptoms and responses on standardized non-cognitive psychological measures. For this reason, it is important to include an assessment of symptom validity when non-cognitive psychological measures are administered.

• Evidence of inconsistent self-report based on an assessment of symptom validity is cause for concern with regard to self-reported symptoms but does not provide information about whether or not the individual is, in fact, disabled. A lack of validity on symptom validity testing alone is insufficient grounds for denying a disability claim, although additional information would be required to assess the applicants’ allegation of disability.

• All non-cognitive psychological assessments should include a statement of evidence of the validity of the results, which could include symptom validity test results, analysis of internal data consistency (e.g., item response theory), and other corroborative evidence as well as discussion of the test norms relative to the individual being assessed.
• For cases in which validation is not achieved, SSA should pursue additional evidence of the applicant’s allegation.

The committee intends standardized non-cognitive psychological tests to include measures of behavior, affect, personality, and psychopathology. By objective medical evidence in this and the following recommendation, the committee means medical signs and/or laboratory or test results that constitute clear objective medical evidence of a significant mental disorder and related functional impairment of sufficient severity to make a disability determination. An example would be a severe brain injury associated with significant functional deficits (e.g., minimally conscious state). By longitudinal medical records the committee means a documented history of a significant mental disorder or a chronic condition such as chronic idiopathic pain or multisystem illness and related functional impairment of sufficient severity and duration to make a disability determination. An example would be a well-documented history of repeated hospitalizations and treatments for a diagnosed mental disorder, such as an affective or personality disorder.

The committee intends the “statement of evidence of the validity of the results” specified in this and the following recommendation to reflect objective evidence that goes beyond the clinical opinion of the examiner. In addition to analysis of the results of SVTs or PVTs administered at the time of the testing and analysis of internal data consistency, evidence could include a pattern of test results that is inconsistent with the alleged condition, observed behavior, documented history, and the like. It is important to note that a finding of inconsistency between the test results and the areas specified is more informative than a finding of consistency would be.

The committee’s recommendation here and in the following recommendation that SSA “pursue additional evidence of the applicant’s allegation” for cases in which validation is not achieved means that the test results in those cases are an insufficient basis to make a determination regarding disability status.

STANDARDIZED COGNITIVE TESTS AND PERFORMANCE VALIDITY TESTS

The following conclusions and recommendation pertain specifically to the use of standardized cognitive tests and associated PVTs in SSA disability determinations:

• Standardized cognitive test results are essential to the determination of all cases in which an applicant’s allegation of cognitive impairment is not accompanied by objective medical evidence.
• The results of cognitive tests are affected by the effort put forth by the test-taker. If an individual has not given his or her best effort in taking the test, the results will not provide an accurate picture of the person’s neuropsychological or cognitive functioning. Performance validity indicators, which include PVTs, analysis of internal data consistency, and other corroborative evidence, help the evaluator to interpret the validity of an individual’s neuropsychological or cognitive test results. For this reason, it is important to include an assessment of performance validity at the time cognitive testing is administered. It also is important that validity be assessed throughout the cognitive evaluation.
• A PVT only provides information about the validity of an individual’s cognitive test results that are obtained during the same evaluation. Evidence of invalid performance based on PVT results pertains only to the cognitive test results obtained and does not provide information about whether or not the individual is, in fact, disabled. A lack of validity on performance validity testing alone is insufficient grounds for denying a disability claim. In such cases,

• All cognitive evaluations should include a statement of evidence of the validity of the results, which could include performance validity test results, analysis of internal data consistency (e.g., item response theory), and other corroborative evidence as well as discussion of the test norms relative to the individual being assessed.
• For cases in which validation is not achieved, SSA should pursue additional evidence of the applicant’s allegation.

QUALIFICATIONS FOR TEST ADMINISTRATION AND INTERPRETATION

The committee reached the following conclusions and recommendation about the qualifications for the administration and interpretation of standardized psychological tests:

• Use of standardized procedures for the administration of standardized non-cognitive and cognitive psychological tests enables application of normative data to the individual being evaluated. Without standardized administration, the test-taker’s performance may not accurately reflect his or her ability. It is important that any person administering cognitive or neuropsychological tests be well trained in the administration protocols for those particular tests, possess the interpersonal skills necessary to build rapport with the test-taker, and understand important psychometric properties, including validity and reliability, as well as factors that could emerge during testing to place either at risk.
• Interpretation of standardized psychological test results is more than a report of the standardized test scores; it requires assigning meaning to the scores within the individual context of the specific examinee. As such, interpretation of test results requires a higher level of clinical training than does the administration alone of some psychological tests.
• Licensed psychologists and neuropsychologists are the specialists qualified to interpret the results of most standardized psychological and neuropsychological tests. Under close supervision and direction

• of licensed psychologists and neuropsychologists, it is standard practice for psychometrists or technicians with specialized training to administer and score tests. Test manuals specify the qualifications necessary for administration, scoring, and interpretation of the test or measure.

• It is important as well that the individual responsible for making the disability determination (disability examiner or administrative law judge) have the training and experience to understand and evaluate the report provided by the psychologist or neuropsychologist.

• “Qualified” means that the specialist must be currently licensed or certified to administer, score, and interpret psychological tests and have the training and experience to administer the test and interpret the results.
• This recommendation applies not only to standardized psychological testing that may be ordered in the course of a disability evaluation, but also to standardized psychological testing already in an applicant’s medical evidence of record if the results are considered as part of the disability determination.

ECONOMIC CONSIDERATIONS

The committee concluded the following with respect to the complex economic considerations raised by increased systematic use of standardized psychological testing by SSA as recommended:

• The average cost of testing services varies by the type of testing (e.g., psychological, neuropsychological), by the type of provider (e.g., psychologist or physician, technician), and by geographical area. The variation in pricing implies that the expected costs to SSA of requiring psychological testing will depend on exactly which tests are required, the qualifications mandated for testing providers, and the geographical location of the providers most in demand.
• Estimating the exact cost of broad use of psychological testing by SSA will require more detailed data on the exact implementation strategy. To fully measure the potential costs, it is likely that SSA will need to pilot the use of testing and the costs associated with it.

• At present, there do not appear to be any independently conducted studies regarding the accuracy of the disability determination process as implemented by DDS offices. Some published estimates of billions of dollars in potential cost savings to SSA associated with the use of symptom validity testing and performance validity testing are based on assumptions that if violated would substantially lower the estimated cost savings. Potential cost savings associated with testing vary considerably based on the assumptions about who it is applied to and how many individuals it detects and thus rejects for disability benefits.
• A full financial cost-benefit analysis of psychological testing will require SSA to collect additional data both before and after the implementation of the recommendations of this report.

EVALUATION AND RESEARCH

Based on its examination of the literature and dialogues with experts in a variety of areas, including psychological and neuropsychological testing, performance validity testing and symptom validity testing, and the disability evaluation process both within SSA and in other arenas, the committee recognizes many questions remain with regard to the use of standardized psychological testing in the disability determination process.

As part of its assessment of the use of standardized psychological tests for the disability evaluation process, the committee was asked to discuss the costs and cost-effectiveness of requiring a single test or a combination of tests. This report provides an initial framework for evaluating the economic costs and highlights the types of data that will be needed to accurately determine the financial impact of implementing the committee’s first two recommendations. The following conclusions and recommendation relate to this enterprise.

• Accurate assessments of the net financial impact of psychological testing as recommended by the committee will require information on the current accuracy of DDS decisions and how the accuracy is affected by the increased use of standardized psychological testing.
• The absence of data on the rates of false positives and false negatives in current SSA disability determinations precludes any assessment of their accuracy and consistency.
• There currently is great variability in allowance rates for both SSI and SSDI among states that is not fully accounted for by differences in the populations of applicants. There also is great variability in the disability determination appeal rulings among administrative

• law judges within and across states. Although it is not possible to know definitively whether the large share of unexplained variation in state filing, award, and allowance rates is driven by variability in the federal disability determination process, there is some evidence that states differ in how they manage claims.

• In light of this unexplained variability, systematic use of standardized psychological testing as recommended by the committee is expected to improve the accuracy and consistency of disability determinations.

• Accuracy refers to the rates of false negatives and false positives in SSA’s disability determinations.
• Consistency means that adjudicators presented with the same evidence for comparable cases come to the same conclusion.

Recognizing that the costs and benefits of implementing the committee’s recommendations go beyond the financial, the committee recommends that SSA evaluate the effect of implementing the committee’s recommendations on its disability determination process using a number of different measures.

• Number of backlogged cases;
• Efficiency of throughput or time to determination;
• Number of requests for appeals;
• Adherence to recommended evaluations;
• Effect on accuracy and consistency of disability determinations; and
• Effect on state-to-state variation in disability allowance rates and on appeal rulings among administrative law judges.

Over the course of the project, the committee identified two areas in particular in which it expects that the results of further research would help to inform disability determination processes as indicated in the following conclusions and recommendation.

• Additional research is needed on the use of SVTs and PVTs in populations representative of the pool of disability applicants, including in terms of gender, ethnicity, race, primary language, educational level, medical condition, and the like. In particular, additional research on the development of appropriate criterion or cut-off scores for PVTs and SVTs in these populations for the purposes of disability evaluation would be beneficial.
• The committee’s task was to evaluate the value of psychological testing in the disability determination process, as reflected in the foregoing recommendations. However, the committee recognizes that just as systematic use of standardized psychological testing is expected to improve the accuracy and consistency of disability determinations for applicants who allege cognitive impairment or whose allegation of functional impairment is based solely on self-report, the use of other standardized assessment tools also may be expected to improve the accuracy of disability determinations. The value of standardized assessment tools, including psychological tests, to assessments of individuals’ work-related functional capacity is an area that would benefit from further research.

• The effects of standardized psychological testing on the accuracy and consistency of disability determinations;
• The use of performance validity tests and symptom validity tests with disability applicants; and
• The use of psychological tests, including performance validity tests and symptom validity tests, in different populations with regard to fairness for members of all gender, ethnic, racial, language, educational levels, and other protected groups.

This page intentionally left blank.