Review of the AHRQ Report

In responding to its Statement of Task, the committee reviewed the Patient Safety and Quality Improvement Act of 2005 (PSQIA) (see selected language in Box 2) in order to better understand the Act and the intention behind the requested report to Congress.

In light of the expressed congressional intent of the PSQIA to establish new entities charged with the collection and analysis of multi-organizational regional and national data, the committee focused significant attention on reviewing the Agency for Healthcare Research and Quality (AHRQ) draft report’s coverage of the use of patient safety organizations (PSOs) and the Network of Patient Safety Databases (NPSD) to improve patient safety. The committee finds that the draft report could be strengthened significantly in its review of the barriers and incentives to PSOs and the NPSD. What follows are the committee’s responses to the questions about additional strategies for improving patient safety (within its Statement of Work) in the context of the legislative intent. These answers are not exhaustive, but rather highlight some of the more important strategies that would improve the report’s final version.

SECTION 1

Additional effective strategies or practices for reducing medical errors and increasing patient safety not mentioned in the draft report.

Since the enactment of the PSQIA, more than 90 PSOs have been formed (AHRQ, 2021). Unfortunately, far fewer have collected and contributed data using the required Common Formats to the NPSD than envisioned in the PSQIA. The committee suggests several strategies that could improve the effectiveness of PSOs (and their data-submission rates) and the value of data aggregation from multiple organizations, including automating data collection from electronic health records (EHRs), facilitating the use of Common Formats or models for that data, using artificial intelligence to derive insights from the aggregate data, and clarifying the legal protections surrounding the submission of data to PSOs.

Addressing Challenges to Data Aggregation Faced by PSOs and Providers

To address additional strategies that would help PSOs become more effective and improve patient safety at the regional and national levels, the committee sought to better understand the challenges PSOs face in accomplishing their mission. In September 2019, the U.S. Department of Health and Human Services’ (HHS’s) Office of Inspector General (OIG) issued a report titled Patient Safety Organizations: Hospital Participation, Value and Challenges, which provides a summary of the important relationships among health care providers, PSOs, and the NPSD, depicted in Figure 1. Only the first step of submitting patient safety work product (PSWP) to PSOs has taken place to any substantial degree, and lessons from aggregate data analysis have yet to emerge. In addition to providing a snapshot of the current state of PSOs and the NPSD, the OIG report highlighted challenges they face and provided three sets of PSO-related recommendations to AHRQ focusing on actionable items that AHRQ could carry out to better support and promote the PSO program. As noted in Appendix B of the OIG report (OIG, 2019), AHRQ agreed with the majority of the recommendations. It would be helpful for AHRQ to include in its final report to Congress an update on the actions taken in response to the OIG recommendations and the specific strategies developed to address the barriers cited.

AHRQ’s PSO program currently provides technical support to 94 active, certified PSOs in the United States. These PSOs are private organizations that work alongside health care providers and strive to improve patient safety and quality of health care delivery. A critical part

of this relationship is the collection and analysis of PSWPs (OIG, 2019). Although 59 percent of acute care hospitals are working with a PSO (OIG, 2019), it is not clear whether PSOs are primarily acting as a consultant to analyze the in-house data on behalf of the organizations or are using their capability to combine data from multiple organizations to generate cross-organizational benchmarks and systemic learnings as Congress intended. If cross-organizational work is being done, it is not being clearly communicated because the majority of hospitals that did not use PSOs felt that these services were duplicated by other patient-safety programs. Contracting with external consultants to assist with an organization’s patient-safety efforts does not require new legislation, but collection and analysis of data from multiple organizations in a protected environment for learning does. Understanding the extent to which multi-organizational data are being used would help the readers of the AHRQ report understand whether and how PSOs are leveraging their PSQIA-enabled data protection permissions to improve safety. The committee believes that strategies to demonstrate the value of aggregate data analysis and communication of that value to potential clients would be productive.

Unfortunately, despite the number of active PSOs, the vast majority of data in the NPSD come from only three PSOs (OIG, 2019). The small number of contributors limits the generalizability of the results. This creates a chicken-and-egg dilemma. In order to facilitate greater participation, clear benefit needs to be demonstrated. Yet, there are barriers to the submission of data for which compelling benefits would serve as motivation to overcome the challenges. Instead of forgoing the request to submit data using Common Formats, the committee believes that AHRQ should explore strategies to reduce the burden of reporting and at the same time increase the benefit to the reporter for doing so. The committee believes there are multiple strategies to break the logjam that could be addressed in the AHRQ report, such as interfacing with EHRs, applying artificial intelligence methods to harvest learnings from the aggregate data, and clarifying the legal protections surrounding the submission of data.

EHRs

The widespread availability of EHR systems and clinical databases offers new opportunities to use electronically captured data to reduce medical errors and improve patient safety. Assessing the value of EHRs in improving patient safety could also benefit from the availability of multi-organizational data contained in PSOs, as noted in the Institute of Medicine (IOM) report Health IT and Patient Safety: Building Safer Systems for Better Care (IOM, 2012). Additional strategies to leverage extant data and reduce the manual burden of generating reports to PSOs would increase the value of data-reporting activities and reduce the costs and burden required for submission. The use of Common Formats for data elements or common data models is key to successful aggregation of data from multiple organizations. The OIG report identified several barriers to the collection and transmission of the data in Common Formats. Because the aggregation function was central to the PSQIA, the AHRQ report could consider addressing the barriers to carrying out the aggregation function and identifying strategies to overcome them. Tools to assist in mapping EHR data to Common Formats may lower the burden of submitting data in Common Formats.

In addition to accessing EHR data for patient safety reporting, a more automated way of submitting data for real-time analysis would further increase the value to reporting health care organizations. In a 2018 study, researchers worked with a PSO and its member systems to develop a patient-safety management system using real-time data from leading commercial

EHRs (Classen et al., 2018). Data from multiple organizations were combined and analyzed within the PSO, and a patient-safety predictive score was generated. This score was tested in two pilot hospital programs. In the pilot programs, a substantially higher rate of safety events was identified and the safety-risk scores were able to predict specific adverse outcomes (e.g., inpatient mortality or readmission). This study demonstrated that real-time data from commercial EHR systems could be used to improve patient safety and highlighted the need to address barriers to data access, expertise, and standardization in order to use the full potential of EHRs in patient safety. This is an important example of how aggregate data can be collected without relying on labor-intensive human reports and derived models can be used to prevent patientsafety events or reduce harm. This example of bi-directional transmission (automatic reporting to PSOs and return of analyzed results to practicing clinicians) with real-time benefit to frontline clinicians is something that was not possible in 2005 when PSQIA was enacted but is now possible subsequent to the Health Information Technology for Economic and Clinical Health Act.. induced adoption of EHRs. Additional strategies to facilitate real-time connections of PSOs to client EHRs and return predictions of patient-safety risks could be studied and exploited as proactive measures to prevent or mitigate harm.

Artificial Intelligence Learning

Now that large volumes of clinical data are available electronically, modern data analytical methods can be applied to large datasets. Artificial intelligence has been employed in health care in many settings (Bates et al., 2020; Pombo et al., 2014; Rickards et al., 2014). Machine learning could be used for the NPSD data to better identify patient-safety incidents and resultant harms and to determine whether the approaches or policies used in different organizations are associated with increased or decreased adverse events (AEs). Natural language processing can help interpret text documents and glean insights from the text (Dandala et al., 2020). Strategies to leverage artificial intelligence to extract insights about patient-safety events and mitigation strategies would enhance the value of aggregate data collected from health care organizations by PSOs.

Legal Protections

Although health care organizations may understand that contributing data about patientsafety events may advance learning about patient safety, they are also concerned about exposing themselves to legal risks. The PSQIA provides legal protection for health care organizations reporting PSWPs, but previous challenges found certain information was not protected (OIG, 2019). The ensuing confusion about the extent and enforceability of data protection provisions has become a significant barrier to PSO effectiveness. The draft report could better respond to the confusion by developing strategies to clarify the legal protections for data submitted to PSOs. Where the protections are insufficient to convince health care organizations to participate, further strategies to address those concerns may require regulatory or legislative action.

SECTION 2

Significant additional or new evidence regarding the effectiveness of particular strategies or practices for reducing medical errors and increasing patient safety.

The committee finds that new evidence and updates to previously referenced evidence could be incorporated into the AHRQ draft report to make it a more useful product to Congress. This includes up-to-date assessments of PSOs and the NPSD and examples of successful data aggregation in other national patient safety programs, such as the Veterans Health Administration (VHA) National Center for Patient Safety (NCPS), the U.S. Department of Veterans Affairs (VA) Surgical Quality Improvement Program (VASQIP), the U.S. Food and Drug Administration (FDA) Adverse Event Reporting System (FAERS), the FDA Sentinel Initiative, and the Centers for Disease Control and Prevention (CDC) Vaccine Adverse Event Reporting System (VAERS). Although the draft report emphasizes the success of the Comprehensive Unit-Based Safety Program (CUSP), a more balanced review that addresses both strengths and weaknesses would be helpful to understanding the value of the program in different settings.

AHRQ Assessments of the Current State of PSOs and the NPSD

The main rationale for the PSQIA was to facilitate the identification of major patient-safety gaps and to develop strategies to prevent or mitigate harm. The committee suggests that the AHRQ report to Congress include an assessment of the degree to which these functions have been achieved, areas in which they have not, and steps that could be taken to address any barriers that are identified. Much of this information is contained in the 2019 OIG report previously referenced.

In its assessment of these functions, the committee suggests that AHRQ include a section in the report on the ways in which health care organizations have benefited from PSO data. The PSQIA states that PSOs should collect data “in a standardized manner that permits valid comparisons of similar cases among similar providers” (Section 299b-24-b.1.F). The current AHRQ draft report does not provide enough information on whether this outcome has been accomplished. If it has not, strategies for addressing the shortfall should be included in the report. In addition, AHRQ should consider the following questions around PSO data and their usage. Although it is not in this committee’s purview to provide answers to these questions, the answers to them would inform effective strategies for improving patient safety.

• What are the characteristics of participating hospitals? Does hospital size affect participation rates? Are rural or underserved hospitals less likely to participate?
• How have data been analyzed and compared among providers?
• Has the requirement for the use of Common Formats helped or hindered this process?
• Have PSOs been able to successfully identify themes in patient safety gaps across different settings and providers?
• Is there evidence of additional value to PSO-collected patient safety data beyond that analyzed by individual organizations? If so, this is an important patient safety strategy not currently addressed in the draft report that should be added. If not, the report should address reasons this strategy has not yet fulfilled its promise.

In addition to PSOs, the NPSD is a crucial component of the safety strategy laid out in the PSQIA. Consequently, the committee suggests that the AHRQ report include an assessment of its performance to date. To what extent has having a national database been an effective patient-safety strategy? The following are suggested NPSD assessment questions that AHRQ should consider addressing in its final draft:

• Has the NPSD been an effective patient-safety strategy?
• What types of analyses have been performed on these data? What reports have been generated from NPSD data?
• How have these reports been used?

Furthermore, a component of the NPSD performance review should include an assessment of the quality of the NPSD database itself. Specific questions for this portion could include the following:

• Are the data representative of the U.S. population and provider organizations?
• Does the database include data from an appropriate mix of locations, specialties, providers, and populations?
• Is the database volume sufficient to generate insights? How is the quality and completeness of the data assessed by PSOs?

It would also be useful for AHRQ to include a comparison among PSOs and other similar safety bodies in different fields, such as the National Aeronautics and Space Administration’s Aviation Safety Reporting System (ASRS) created in 1976 (ASRS, 2021). Because the PSO structure was modeled after this voluntary system, comparing their outcomes could provide useful insights into potential improvements.

The AHRQ report could also examine other national patient safety programs, such as those highlighted in the following sections, for strategies they employ to learn from aggregate data analysis.

NCPS

The VA created the NCPS in 1998 as a system-oriented approach to reducing AEs and close calls. It was pilot tested from 1999 to 2000, and rolled out to the entire VHA system (VHA, 2021). The primary components of the NCPS include “developing and providing patient safety programs and delivering standardized tools, methods, and initiatives to all VA facilities” (Heget et al., 2002). The goal of the center was to develop a culture that avoids punishing people for reporting problems by emphasizing system-based learning, actively seeking out close-call events, and using a novel root cause analysis process using multi-disciplinary investigatory teams (Heget et al., 2002; Neily et al., 2003). AHRQ may wish to explore how the VA has used its large amounts of patient safety data to enhance veteran safety as this may help inform efforts in non-VA clinical settings. Including such information will likely be useful to readers of the revised AHRQ report.

The VA Surgical Quality Improvement Program

VASQIP was established in 1994 as a result of the congressionally mandated National VA Surgical Risk Study to provide a validated tool that could compare and monitor surgical outcomes in VA medical centers (VAMCs). It has become a national program, the National Surgical Quality Improvement Program (NSQIP), led by the American College of Surgeons. The information from the NSQIP can be used for self-assessments and internal reviews by surgical departments. After conducting a study to determine the success of the NSQIP in its first 4 years of use, Khuri and colleagues (1998) concluded that it contributed to lower rates in both morbidity and mortality across the participating VAMCs. Its success can be attributed to clinical nurse reviewers who are trained in data collection and quality assurance (Lipshy et al., 2021), as well as the VHA’s uniform data collection system, which allows access to uniform administrative and clinical information entered by surgical nurse reviewers.

As mentioned in Section 1, natural language processing can help interpret text documents and glean insights from the text. It can be used to identify patient-safety incidents, understand treatments and mitigation, and identify best practices. In a cross-sectional study at VAMCs, researchers sought to compare the difference between natural language processing analysis and patient safety indicators (PSIs) identified from administrative patient discharge information (Murff et al., 2011). They looked at inpatient surgeries in six VAMCs over 8 years to compare the two methods. The analyzed data came from a random sample of VASQIP-approved surgical inpatient admissions. The researchers concluded that natural language processing-based analyses had higher rates of sensitivity and lower specificity overall than the administrative PSIs and therefore were likely to lead to more event detection but with a higher false positive rate.

FDA Methods

FDA provides another example of sophisticated methods of using voluntarily reported safety data effectively and making them accessible to the broader community to improve safety.

FDA Sentinel Initiative

Created in 2008 through a congressional mandate, the current form of the FDA Sentinel Initiative includes electronic health care data from 16 data partners that are used to identify patterns of medication use and outcomes that signal new or unanticipated medication-related safety events. Data are formatted locally in the Sentinel Common Data Model and submitted to the Sentinel Distributed Database, which currently includes more than 350 million unique patient identifiers, 788 million person-years of data, and 14.8 billion pharmacy dispensings (Sentinel Initiative, 2021). The Active Risk Identification and Analysis system has conducted dozens of safety analyses of specific drugs or interactions using these data, generating important safety evidence that has changed practice. For instance, a large number of serious and fatal bleeding events were reported to FAERS after approval of dabigatran, a novel anticoagulant (FDA, 2015). FDA used Mini-Sentinel (the precursor to Sentinel) data to establish that the risk of bleeding was not in fact significantly higher than with warfarin (for which, as an older drug with established adverse effects, people were less likely to voluntarily report bleeding). FDA released these findings in a safety report to reassure providers (FDA, 2014) and then conducted a larger study of Medicare patients to confirm the results (FDA, 2018).

VAERS

A collaborative effort of FDA and CDC, VAERS is another voluntary AE reporting system established in 1990 by congressional mandate. Reports can be submitted by anyone, including manufacturers, clinicians, and the general public, and are monitored for signals of serious safety concerns by FDA and CDC staff in a similar fashion to FAERS. In 2019, more than 48,000 events were reported. As with FAERS, data are publicly available and can be searched or downloaded by the public (CDC, 2020). VAERS reports have led to important practice changes. For instance, the first rotavirus vaccine (RotaShield) was approved on August 31, 1998. By July 16, 1999, VAERS had received 15 reports of intussusception in vaccine recipients. This prompted CDC to conduct a case–control study, which demonstrated a strong relationship with intussusception, and the vaccine was withdrawn from the market (Foster, 2007). Subsequent trials of other rotavirus vaccines were required to study intussusception explicitly to ensure safety.

Balancing the CUSP Review with Additional Evidence

While AHRQ’s draft report focuses heavily on CUSP and its successes, review of its weaknesses as well as its strengths would be helpful to the reader as the literature on CUSP is more nuanced than the draft report implies. The draft report presents CUSP as a clear success, but it should include examples where it has been less effective as well. Specific examples, discussed in more detail below, include the “Matching Michigan” program in the United Kingdom, the success that the VA had in reducing central line–associated bloodstream infections (CLABSIs) without using CUSP, and the ineffectiveness of CUSP in reducing CLABSI and catheter-associated urinary tract infection (CAUTI) rates in U.S. intensive care units (ICUs) with elevated infection rates.

• The “Matching Michigan” program ran in the United Kingdom from 2009 to 2011, using the interventions and structure of CUSP that were implemented in Michigan (Bion et al., 2012; Pronovost et al., 2006). Based on the reported successes of CUSP,

• the United Kingdom’s National Patient Safety Agency decided in 2008 to implement it in an effort to reduce all CLABSI rates in the country to (at least) the mean reported by CUSP (1.4 per 1,000 central venous catheter-patient days). They used a three-pronged approach: ensuring uniform use of evidence-based methods, interventions at the system and cultural level, and a standardized reporting system for all CLABSIs. The Matching Michigan program found that the CUSP interventions had no effect on declining CLABSI rates. Although the rates did drop to the desired levels, they concluded the declines were the result of extant trends, and not due to the interventions, because the rates of the downward trends did not increase after the intervention began.
• The VHA was able to achieve reduced CLABSI rates without using CUSP. Inspired by the Institute for Healthcare Improvement (IHI) “100,000 Lives” campaign, the VHA implemented a system-wide (174 ICUs) program to reduce CLABSIs and ventilator-associated pneumonia (Render et al., 2011). It designed the intervention around a centralized infrastructure called the Inpatient Evaluation Center that supported evidence-based methods to reduce CLABSIs, supported by “leadership, benchmarked feedback, learning tools and selective mentoring” (Render et al., 2011, p. 725). From 2006 to 2009, the intervention reduced CLABSI rates from 3.8 to 1.8 per 1,000 line days (p <0.01). Incidence rate ratios also declined from year to year. The data showed a strong correlation between bundle adherence (grouping prevention methods such as handwashing and maximal barriers) and CLABSI rates (r = 0.81).
• A recently published CUSP-based approach did not reduce rates of CLABSIs or CAUTIs in U.S. ICUs with persistently elevated rates. A national project team and state hospital associations implemented an adapted CUSP program, which included on-demand video modules and live virtual instruction (Meddings et al., 2020). The program content consisted of a two-tiered CUSP approach, including technical and socio-adaptive factors designed to prevent catheter infections. Units reported infection rates for CLABSIs and CAUTIs and catheter use data to the program for analysis and evaluation. This study recruited 220 hospitals (366 total ICUs) from 16 states and Puerto Rico, and 274 ICUs provided complete data for analysis. The final analysis of all data showed no significant reduction in either CLABSIs (incidence rate ratio [IRR] = 0.75, 95% confidence interval [CI] 0.52–1.08, p = 0.13) or CAUTIs (IRR = 0.79, 95% CI 0.59–1.06, p = 0.12). Central line use did not change (adjusted IRR = 0.97, 95% CI 0.93–1.00, p = 0.09), nor did urinary catheter use (adjusted IRR = 0.98, 95% CI 0.95–1.01, p = 0.14).

Discussion of these additional assessments and provision of approaches to help hospitals with higher infection rates would be useful to include in the AHRQ report.

SECTION 3

Methods for encouraging effective adoption and sustained implementation of effective strategies for reducing medical errors and increasing patient safety.

Despite the benefits of an intervention’s end solution, user adoption and the sustainability of that intervention often remain a challenge. Implementation science studies the means by which interventions are carried out in practice and develops strategies to improve end-user adoption, effectiveness, and sustainability. The AHRQ draft would be strengthened by discussion of specific implementation strategies relevant to adoption and sustainability of the NPSD, such as a focus on the value to reporting organizations (e.g., incentives, such as better analytics in NPSD outputs) or efforts to reduce reporting burden, which are important for the country to realize the benefits of the original PSQIA objectives.

Implementation Strategies

Implementation science is “the scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice” (Eccles and Mittman, 2006, paragraph 1). It has been useful in enhancing research in patient safety, especially in infection prevention. A systematic review by Braithwaite and colleagues identified eight success factors of implementation: “preparing for change, capacity for implementation-people, capacity for implementation-setting, types of implementation, resources, leverage, desirable implementation enabling features, and sustainability” (Braithwaite et al., 2014, p. 321). In another study, Damschroder and colleagues (2009) distinguished between descriptive implementation models and action-oriented ones. Finally, the Safety Assurance Factors for EHR Resilience (SAFER) guides may also offer direction as these tools include recommended practices that can help health care organizations optimize their use of EHRs in a safe manner (AHA, 2021). The AHRQ draft report may want to include additional details, similar to these studies and approaches, about which implementation models and approaches would be most appropriate for PSOs to use, and under what circumstances.

For example, the OIG report notes that organizations not involved with PSOs typically expected little additional value in engagement and therefore were not interested in even beginning the process. A critical factor in the success of voluntary reporting systems is the perception of value—to both the reporter and the broader community. Thus, the benefit to the individuals expending effort to achieve meaningful participation in voluntary programs should be made obvious. The Quality Implementation Framework is a useful implementation science framework that devotes substantial attention to Phase 1 of any intervention, which includes assessing needs, fit, and capacity/readiness (Meyers et al., 2012). Taking the time to develop buy-in, make the value of the intervention clear, and understand institutional needs is critical to successful adoption. The AHRQ report should explore strategies for communicating the value of PSOs for organizations and reporting individuals to increase participation in the existing programs. A strategy should target all of the stakeholders who would benefit from PSOs’ aggregate data functions, including health systems executives, patient safety executives, medical group leadership, and frontline clinicians.

The OIG report also notes numerous barriers to data submission. Strategies to increase the use of Common Formats should be undertaken to facilitate the aggregation and analysis of

multi-institutional data. There are multiple barriers to using Common Formats. Currently, Common Formats only apply to acute care hospitals, community pharmacies, and skilled nursing facilities. Strategies to expand the use of Common Formats by other organizations should be undertaken to broaden the scope of captured data related to patient-safety events.

Finally, effective implementation science strategies may need to be different for high-versus low-performing systems. Effective conceptual models are needed to improve care in low-performing health systems.

Behavioral Economics Strategies

Behavioral economics concerns itself with understanding how and why individuals make decisions. Some of these strategies can be leveraged to increase appreciation of the value that PSOs and the NPSD could provide, encouraging organizations to engage. Behavioral economists have found, for instance, that people are more motivated by loss aversion than by potential gain. Messaging focused on potential losses from missed safety opportunities that PSOs or the NPSD can help mitigate might therefore be helpful in encouraging effective adoption. By contrast, some reluctance to participate in the NPSD stems from a concern that reporting might not be legally protected, making reporting appear risky. Stakeholders have asked for clarification of the legal protections afforded to data submitted to PSOs (OIG, 2019). AHRQ should work with the Office for Civil Rights, which HHS has designated as the office responsible for interpreting and enforcing the legal protections, to understand the concerns of stakeholders and respond with clarifications and guidance. Clarification of the legal protections afforded data submitted to PSOs is an essential part of the communications strategy to stakeholders. In addition, a description of the privacy preserving procedures for de-identifying the data before their transfer to the NPSD should be included. This will strengthen the perception of PSOs’ stability, trustworthiness, and sustaining value and reduce the perception of risk or potential for loss through participation.

Implementing appropriate incentives is another important strategy for adoption and sustainability. In the aviation industry, as an incentive for more complete and timely reporting, the Federal Aviation Administration waives fines and penalties for unintentional procedural violations if a report is submitted to the ASRS within a specified period (FAA, 2007). This incentive motivates individuals to spend the time and effort to submit a report in the format requested. A similar incentive does not exist for the NPSD. AHRQ could explore strategies to develop payer (e.g., U.S. Centers for Medicare & Medicaid Services) incentives to encourage participation in PSO programs. To participate in such an incentive effort, these specific PSO programs should be certified to accept data using Common Formats and to transmit relevant data to the NPSD using Common Formats.

SECTION 4

Technical comments and corrections to the AHRQ draft report.

The committee has a number of technical comments that, if addressed, could strengthen the lessons learned from the past and position PSOs and the NPSD for even more impactful successes. The current draft report focuses primarily on the patient-safety activities sponsored by

AHRQ that are well documented in the MHS series of reports.¹ The committee proposes that the approach to these evidence sources could be improved on and that the breadth of the activities addressed in the draft report could be expanded to include other federal and international programs that may be useful in developing new strategies to enhance the success of the program.

As part of its description of patient-safety strategies in Chapter 2, AHRQ should consider including other high-profile patient safety programs that offer important lessons, in addition to the patient safety programs in which AHRQ is already involved. In the context of the PSQIA, it would be interesting to consider how these programs (and others already mentioned in the draft report) could benefit from the NPSD. Are there collaboration or partnership opportunities for the use of multi-organizational aggregate data?

Consider Including Additional High-Profile National Patient Safety Programs and Initiatives

• The CDC Epicenter Program, a research program in which CDC collaborates closely with academic investigators, has provided the basis for many of the evidence-based practices currently used to prevent health care–associated infection, including CLABSIs (CDC, 2015). Research conducted by investigators supported by the CDC Epicenter Program has formed the basis for many of the recommendations used to prevent infection, including universal decolonization to prevent MRSA (which was appropriately profiled in Chapter 2 of the AHRQ draft report) and approaches to reduce CLABSIs (Warren et al., 2003, 2004, 2006) and ventilator-associated pneumonia (Babcock et al., 2004; Huang et al., 2013; Zack et al., 2002). The CDC Healthcare Infection Control Practices Advisory Committee also provides evidence-based guidelines and recommendations on how to prevent common health care–associated infections, including CLABSIs and CAUTIs (CDC, 2021).
• The IHI is a leading nongovernmental agency in enhancing patient safety. The national “100,000 Lives Campaign” is an example of an IHI effort designed to improve patient safety (Baehrend, 2016; Berwick et al., 2006). The campaign’s interventions targeted six areas of patient safety for improvement: rapid response teams, improved care for acute myocardial infarction, medication reconciliation, prevention of central line infections, prevention of surgical site infections, and prevention of ventilator-associated pneumonia (Baehrend, 2016; Berwick et al., 2006).
• The Joint Commission is the not-for-profit organization responsible for accreditation in health care. This organization contributes to patient safety by ensuring that health care organizations meet certain levels of quality and producing an annual National Patient Safety Goals report (The Joint Commission, 2021). This report gathers information about emerging patient-safety concerns across the health care spectrum. The commission also creates standards by which organizations can objectively measure, assess, and improve performance.

___________________

¹ The three MHS reports can be found at https://www.ahrq.gov/research/findings/making-healthcare-safer/index.html (accessed March 21, 2021).

Making Healthcare Safer Reports

Chapter 2 (Strategies for Reducing Medical Errors and Increasing Patient Safety) of the AHRQ draft report summarizes the MHS reports along with brief overviews of systems approaches to patient safety (e.g., James Reason’s “Swiss Cheese Model” of error), human factors engineering, and the role of high-reliability organizations in patient safety. Much of Chapter 2 focused on the MHS reports, which are nicely summarized in Section 2.5 (Existing and Emerging Strategies for Reducing Medical Error and Increasing Patient Safety) and Tables 1 to 28, with special attention paid to the third and most recently released report, Making Healthcare Safer III (MHS III), released March 2020. Of note, MHS III included patient safety literature up until 2018.

The committee acknowledges the tremendous work that went into creating each of the MHS reports, beginning with the first one published in 2001 (Shojania et al., 2001). The initial report, Making Healthcare Safer: A Critical Analysis of Patient Safety Practices, involved a commissioned group of about 40 investigators in various disciplines such as patient safety and evidence-based medicine who identified 83 safety practices that were supported by 70 systematic reviews and 293 additional primary investigations (Shojania et al., 2001, 2002). This effort was led by Dr. Kaveh Shojania and colleagues at the University of California, San Francisco–Stanford University Evidence-Based Practice Center. The initial report generated substantial interest (as evidenced by the high number of copies requested and number of downloads) but was also met with a critique by leading patient safety experts. Specifically, Drs. Lucian Leape, Donald Berwick, and David Bates wrote

Shojania and colleagues responded by writing “In the end, we agree with Leape et al. that the best approach for ensuring patient safety will be one in which the general insistence on evidence does not prevent implementation of practical, low-risk, but understudied interventions that seem likely to work” (Shojania et al., 2002, p. 512).

Subsequent reports (MHS II and MHS III) updated the evidence for these initially identified practices and also included evidence summaries of additional patient safety practices. The draft AHRQ report effectively uses Tables 1 to 28 to concisely summarize this enormous amount of work spanning nearly two decades. There are links to each relevant MHS report based

on the specific practice listed, which total 108 practices. Even though providing a detailed review of the most recent report (MHS III), which spans 1,400-plus pages, is beyond the scope of the present report, the committee identified some strategies AHRQ might use to make its draft report and future MHS reports even more useful to policy makers, frontline providers, and patient safety researchers. Suggestions include

• Make the rationale for the inclusion and exclusion of selected areas explicit. Although the AHRQ draft report indicates scanning multiple reports, several key patient safety issues were not addressed comprehensively or at all. Examples of topic areas that it would be beneficial to address in greater detail include EHR-related issues, clinician–patient ratios, the effect of clinician burnout on patient safety, recommendations for preventing health care device-associated infections, and patient safety related to mental/behavioral health. Future MHS reports ideally should provide a more transparent and systematic approach to selecting topics. Another consideration is whether guidance on topic selection could come from a group of stakeholders that include patients, policy makers, and providers.
• Provide evidence-based guidance to the end user for prioritizing recommendations based on demonstrated impact beyond the developer’s organization. Although the AHRQ draft report provides the rationale for moving away from providing single determinations of the strength of evidence for the patient safety programs covered, the key takeaways in the summary table ES.2 from MHS III could provide concise direction and guidance. Priority patient safety areas across the three MHS reports have shifted, but future MHS reports could better summarize how patient safety programs have evolved over time (and how the recommendations and guidance of the reports may have contributed to positive changes).
• Summarize the key clinical, policy, and practice recommendations at the end of every chapter. Additionally, the committee agreed that AHRQ should consider developing a separate, accessible patient-facing version of the draft report that focuses on what patients and families should know about patient safety.
• Establish a regular schedule for MHS reports. Producing MHS reports on a regular schedule (e.g., every 5 years) rather than at various intervals would allow the patient-safety field and health care policy makers to look forward to updated recommendations and actionable information that can be translated into everyday practice.

International Implementation Examples

Although the committee realizes that other countries’ experiences do not necessarily reflect the U.S. experience regarding patient safety and the creation of PSOs and the NPSD, the committee finds some lessons learned from other countries may be helpful to readers of the final AHRQ report. The World Health Organization (2020) report titled Patient Safety Incident Reporting and Learning Systems: Technical Report and Guidance provides an international perspective on this topic. Additionally, England’s National Reporting and Learning System, perhaps the largest patient safety reporting system in the world, may provide some useful insights on the use of large-scale aggregate data relevant to the NPSD (de Feijter et al., 2012; Howell et al., 2015). Other international patient-safety programs that may provide some lessons

on which the AHRQ report authors could draw to help develop strategies suitable for the United States include

• Norway In 2011, an NCPS Steering Committee worked with the Royal Norwegian Ministry of Health and Care Services to implement a campaign to reduce AE rates in hospitals across the country. They required all acute care hospitals to carry out random reviews of medical records and map AE rates using the same Global Trigger Tool (GTT), which provides a potentially applicable lesson for the United States because the GTT was initially developed and refined by the U.S.-based IHI to standardize quantitative and qualitative medical record data (Classen et al., 2008). The random reviews consisted of 10 random adult patient discharge records that were selected every 2 weeks and assessed for criteria that could indicate AE risk. Each hospital was provided with training on how the GTT should be applied and how the data should be reported nationally (e.g., all national data had to be unidentifiable in order to protect privacy). Hospitals were also able to use a Web-based tool called “Extranet,” which allowed them to analyze their own data run in chart form. After this implementation, AE rates in hospitals decreased significantly. The Ministry of Health and Care Services established a new policy in 2012 that required hospitals to reduce their overall AE rates by 25 percent by 2019 (Braithwaite et al., 2014).
• China The National Patient Safety Incidents Reporting System (NPSIRS) was implemented in 2012, replacing a primarily paper-based system that was hindered by low rates of reporting. In a study carried out by Gao and colleagues (2019), 5 years of patient-safety incidents were analyzed (36,498 incidents). Each of the NPSIRS’s reports, which are mandatory and non-confidential, included administrative details (e.g., time/location/those involved), patient demographics, management measures, assessment of harm, and narrative free-text descriptions. The authors concluded that more work was needed, specifically in the areas of nursing care, surgery, and medication safety. There were also disparities noted between regions and hospitals, suggesting a greater need for culturally tailored reporting systems. The authors suggest that “policymakers and health managers at all levels should dedicate to building an open, transparent, and supportive patient safety culture, especially in hospitals in underdeveloped areas, at lower levels, and with smaller sizes” (Gao et al., 2019, p. 267). Lessons from the Chinese experience could inform U.S. strategies for understanding whether such regional variation occurs in the United States and, if so, what can be done about it.
• Sweden A national law requires the 21 regional city councils to produce annual public patient safety reports (PSRs). Since 2011, the PSRs have summarized the current state of patient safety across the regions so that trends could be monitored and any potential gaps could be addressed. The city councils are also provided with an optional template with guidelines for what to focus on in the reports. In a study of PSRs from 2014, Ridelberg and colleagues (2016) found that reports were characterized in a narrative fashion that covered everything from the structure of the patient safety work (e.g., antibiotic prescription issues of EHRs) to process and result elements. The same study carried out a survey of 222 health care practitioners and found that, overall, the PSRs are viewed as a form of benchmarking between councils that allows them to disseminate best practice examples and encourage ongoing

• discussion. The authors of the AHRQ draft report might assess whether this practice is relevant to the United States, in which the states could function in a manner similar to the city councils do in Sweden.

The foregoing list of technical comments is not exhaustive, but highlights certain main technical areas that could be improved on in the draft report. The committee assumes that AHRQ will address any small-scale edits (e.g., citations, grammar) independently and has not included them in the present report.

CONCLUSION

The current AHRQ draft report focuses on general practices and strategies for improving patient safety already identified through literature reviews from the MHS series of reports. The committee acknowledges the tremendous work that went into each MHS report and also provides some recommendations to further enhance the usefulness of future MHS reports. Considering the PSQIA context for the AHRQ draft report, the committee believes that the report could address the effectiveness of the two key PSQIA-related resources (PSOs and the NPSD) more comprehensively. The need for a national data resource for shared intra- and inter-organizational learning was highlighted in the National Action Plan to Advance Patient Safety (IHI, 2021), which AHRQ co-chairs. The committee suggests several strategies to be considered for inclusion in the final AHRQ report that could improve the effectiveness of PSOs and enhance the value of data aggregated from multiple organizations. These include automating EHR data collection, facilitating use of Common Formats for that data, and using artificial intelligence to derive insights from the aggregate data. Additional evidence regarding effectiveness that could be considered for discussion in the AHRQ draft include up-to-date assessments of PSOs and the NPSD and examples of data aggregation successes in other national patient-safety programs, such as the VHA. Many of these strategies are touched on in the 2019 OIG report Patient Safety Organizations: Hospital Participation, Value, and Challenges, which the committee felt would be a critical resource in updating the AHRQ draft in a way that strengthens the value and impact of PSOs and the NPSD in improving patient safety and reducing medical errors. In addition to the foregoing, the committee’s specific suggestions for additional strategies are listed below:

• Fund demonstration projects and communicate compelling evidence of the benefits of PSOs and the NPSD, both to client organizations and to patient-safety researchers. Highlight the benefits of transparency and the utility of analyzing data from multiple, similar organizations for benchmarking and learning.
• Communicate clear guidance about the legal protections afforded data submitted to PSOs and the NPSD, including privacy protecting procedures. Where protections are deemed insufficient to address health care organizations’ concerns, strategies for encouraging further regulatory or legislative actions may be required.
• Increase the use of Common Formats (e.g., definitions, data elements) as a critical intermediary step to shared learning and benchmarking. Explore inclusion of Common Formats use as part of the certification criteria for PSOs. Explore the use of natural language processing as an adjunct to gathering patient-safety information from free-text fields.
• Expand the coverage of Common Formats into ambulatory care and specialty care.

• Explore incentives and technical strategies to facilitate automated transmission of patient-safety data from EHR systems to PSOs to reduce the costs and burden of reporting. Include attention to data quality and completeness.
• Explore the use of artificial intelligence (e.g., machine learning, natural language processing) to enhance the value of the insights from large, multi-organizational datasets and to potentially reduce the burden of data collection and normalization.
• Explore payer or regulatory incentives for reporting safety data to PSOs in a standardized format as a strategy to increase voluntary reporting that facilitates return benefits to both the reporting organizations and to the systematic enhancement of patient safety for the nation.
• Include results of additional studies related to CUSP in which it may not have been effective as well as studies of approaches that can help struggling hospitals improve patient safety.
• Explore high visibility implementation strategies from other national patient-safety initiatives—such as the VHA NCPS, FAERS, the FDA Sentinel Initiative, VAERS, the CDC Epicenters Program, IHI, and The Joint Commission—and appropriate international examples.
• Update AHRQ’s responses to the OIG report recommendations with any subsequent results.

Creating a continuous learning system, with patient safety as a core tenet, is achievable and enabled by the PSQIA. There have been noteworthy accomplishments in patient safety as summarized in the MHS report series. The committee believes that leveraging PSOs and the NPSD to collect and analyze multi-organizational aggregate data would add a critical set of resources that would help the country realize the patient-safety benefits envisioned in the PSQIA. The country stands at a crossroads—with an opportunity to power patient-safety research and practices in the new data era. The committee hopes this peer review enhances the efforts to bring about that future.