Summary

Monitoring recreational fishing catches is critical for maintaining fishing stocks at appropriate levels. For some fish species, recreational fishing is the major form of fishing activity. At the same time, monitoring recreational fishing is a difficult task. There is no national list of recreational fishers, and although states maintain lists of licensees, their lists are not comprehensive.¹ Fishers may use a variety of locations, many of which are too infrequently used to be good locations for monitoring. Further, while fishing activity is often close to where recreational fishers live, it is not limited to such locations. To deal with these challenges, the Marine Recreational Information Program (MRIP) uses a split strategy: measuring fishing effort (i.e., the frequency of fishing) through one survey and estimating fishing catches and discards though another set of surveys. The results are then combined to estimate the impact on fishing stocks.

Monitoring fishing is the responsibility of both the federal and state governments, and the different governmental levels have different needs: for example, fishing seasons vary depending on the geographic location and the fish species, fish species also vary geographically, and fishing constituencies may vary. Thus, although there are federal surveys of fishing effort, catches, and discards, states sometimes perceive the need to conduct their own more specialized surveys. The federal surveys by themselves are not necessarily designed to meet the needs of the states, while state data are needed by the federal government so that it can develop comprehensive

___________________

¹ Creating a national list was recommended in previous National Academies reports (National Research Council, 2000 and 2006).

statistics. The diversity of approaches presents a challenge for creating a national database, given that it results in differences in how data are collected, processed, and reported.

MRIP responded to the issues arising from the diversity of approaches by creating data quality standards.² These standards both guide federal efforts to collect data and also give states guidance on how to construct their data to obtain federal certification (which is helpful when seeking federal funding) and have their state data included in MRIP data program. Thus, the standards provide a mechanism for promoting high data quality from all sources.

Given the importance of the data quality standards and an increase in state-conducted surveys, combined with complaints about one of the standards (leading to changes by MRIP) and anticipation of Congressional direction, the National Oceanic and Atmospheric Administration (NOAA), which houses MRIP, asked the National Academies to convene a group of experts to perform a peer review of the standards. The specific charge to the panel is presented in Box S-1.

NOAA also asked the panel, as part of its review, to consider three special topics and how they applied to the data quality standards: the rise of non-probability data sources, standards for data integration from different

___________________

² https://www.fisheries.noaa.gov/recreational-fishing-data/recreational-fishing-survey-and-data-standards

sources (e.g., to inform small-area estimation), and the use of artificial intelligence (AI) to identify and incorporate potential administrative and other data sources.

In response, the National Academies convened a panel of six experts in statistics, survey methodology, ocean sciences, and fisheries management. The panel conducted the peer review by hearing presentations from 17 individuals active in fisheries research and management, marine and ocean sciences, and federal statistical policy. The panel also reviewed relevant documentation and research, including MRIP publications and MRIP website, the policies of OMB and other federal agencies, and published research literature (for specifics see the citations in the text and the accompanying references). This report presents the results of the review organized by the seven MRIP data quality standards.

THE PANEL’S REVIEW OF THE STANDARDS

Developing and implementing data quality standards is important, both for MRIP-sponsored surveys and for those surveys conducted by states and regional agencies. When states and regional agencies are involved, sharing a common set of standards becomes especially critical, helping to ensure that the data are compatible and can be aggregated meaningfully.

Thus, MRIP has taken an important step by developing its data quality standards. The topics addressed by each of the seven standards are listed in Box S-2. The standards are each provided verbatim in Chapter 3, with a separate section for each standard.

In this section, the panel’s conclusions and recommendations concerning the standards are summarized briefly.

The standards are well designed. They are consistent with OMB’s Standards and Guidelines for Statistical Surveys, the standards used by federal statistical agencies, and with the standards of other organizations. They are also well accepted by those collecting and using the data. To the extent that this report is critical of the standards, it is largely in suggesting ways of expanding beyond the current standards to cover a broader range of situations.

The panel notes that all seven standards explicitly assume both a survey-based approach (e.g., see Standard 1) and probability sampling. Systematic, high-quality data collection through surveys is the primary tool for supporting the data standards and can be expected to remain the primary tool in the future. The panel also notes the potential for including other types of data using new data-science techniques and statistical workflows (Conclusion 4-2), responding to MRIP’s request that the panel examine some special topics relating to new topics in data collection and analysis. For both federal and state data collections, it would be worthwhile to examine how alternative approaches could be useful as a supplement. A complete conceptual design would incorporate all of these tools and discuss how they might be implemented.

Standard 1: Survey Concepts and Justification

The study panel is supportive of this data standard but believes the text of this standard could benefit from minor improvements. The standard might be refocused to concentrate specifically on the core questions that are intended to be addressed, the content and attributes of the data collected in the survey, and the use of the data once collected. The panel suggests the standard provide definitions of key terms (such as how to measure fishing, where feasible). Now focused primarily on survey research, this standard might be broadened to consider alternative methodologies as a supplement to (but not a replacement for) current survey approaches.

Standard 2: Survey Design

The panel supports this data standard. The panel notes it would be helpful to include how non-sampling error (e.g., error associated with questionnaire design or data collection procedures rather than sampling) could be minimized, such as through pretests and cognitive interviews.

Standard 3: Data Quality

The panel recognizes the central importance of this data standard. Minor improvements would be helpful for Standard 3. The standard requires

the provision of both edited and unedited values, but doing so is contrary to standard practice and could lead to breaches in confidentiality. Other improvements would be to incorporate a total survey error framework for considering both sampling and non-sampling error, and potentially to use artificial intelligence (AI) to reduce the burden on researchers.

Standard 4: Transition Planning

The panel believes Standard 4 is well justified. For all transitions, even including process improvements, the agency responsible for the data collection is responsible for examining whether the change has the potential to affect estimates and for performing transition planning when such potential appears.

Standard 5: Review Procedures

The panel finds this standard is well designed. Standard 5 might be improved further by incorporating the standards set out by the American Association for Public Opinion Research (AAPOR) for the way response rates are calculated (Recommendation 3-2). Further, to promote certification while recognizing the difficulties that some states or regional agencies may have in meeting all standards, MRIP should consider providing for multiple tiers of certification, depending on how completely the standards are met (Recommendation 3-3). This would encourage greater participation in MRIP while also providing guidance to data users on data quality.

Standard 6: Process Improvement

Agencies may develop process improvements both to respond to needs, such as problems in data collection, and to take advantage of opportunities, such as new developments in technology or statistical tools. The current Standard 6 is well designed and reminds agencies of the need to document such improvements. It is important to measure the impact of the improvements on survey estimates and to prioritize the improvements accordingly.

Standard 7: Access and Information Management

The panel recognizes the central importance of this data standard. A key topic examined by the panel was the controversy that appeared over one requirement contained within Standard 7: that estimates with percent standard errors (PSEs) greater than 50 percent should be suppressed as not being sufficiently reliable for publication. Some objections were raised when this standard was first proposed. One objection was that the estimates are

needed for fishery management even if they are not of the desired quality, and another objection was that there was a lack of transparency, with some data users able to access the data using the survey microdata while the public did not have access within the MRIP Query Tool.

The panel concluded that while the suppression of estimates with low precision (consistent with the PSE requirement in MRIP) is a practice often used by federal statistical agencies, it is also one where exceptions have been allowed. In particular, exceptions have been made when such estimates were needed for policy purposes and there was no alternative source of data. The panel also determined that transparency is an important goal, so that a preferable approach is to present the estimates, along with notes about the data limitations, and then let data users make their own determinations about whether and how to use the estimates (Recommendation 3-4).

It would also be helpful to provide a forum in which state agencies can provide feedback on how such reporting policies address their own needs (Recommendation 3-5). The current MRIP practice of publishing estimates along with clear color-coded warnings is an effective means of providing data transparency.

EXPANDING BEYOND THE SEVEN STANDARDS

Some key issues are not specifically tied to a single standard. Instead, they apply to multiple standards or to larger issues not covered in the standards.

Accommodating State Priorities

While the federal government is responsible for regulating ocean fishing in the United States’ exclusive economic zone (3 to 200 miles offshore), states are responsible for regulating ocean fishing generally from the shoreline to 3 miles offshore, as well as fishing in all inland waters. However, some fish migrate across federal and state waters, giving the federal and state governments overlapping jurisdictions. States are also the authorities that issue recreational fishing licenses, making them the best authorities on who is involved in fishing. On the other hand, states sometimes have difficulty obtaining the statistical expertise they need and often face budgetary constraints. Thus, state and federal governments have different but interrelated needs, and they also differ in their resources and capabilities.

MRIP standards are designed to encourage cooperation and coordination, while allowing states flexibility to meet their own needs. By establishing a certification system, the federal government provides a way for the various databases to be compatible with each other, and the offer of help with federal financial assistance provides an incentive for states to obtain

certification. There may be value in promoting greater coordination between the federal and state governments (Conclusion 2-3). There also could be value in promoting greater consistency in how catches are enumerated (Recommendation 2-1).

MRIP already provides technical expertise on surveys and statistics, and the panel recommends that such assistance be expanded to better meet the needs of the states (Recommendation 2-2). States often have limited resources and may benefit from such assistance.

One quality not included in the data standards but often requested by data users is the timeliness of data release. The panel recognizes that data collection and processing are time consuming and that MRIP is partially dependent on other factors, such as the time delay associated with receiving mailed questionnaires. It is possible that both timeliness and data quality could be advanced by such tools as the use of phone apps, statistical modeling, and AI for data processing, and it would be desirable to explore such tools as long as other aspects of data quality could be maintained (Recommendation 2-4).

Enhanced Adoption of Model-Based Statistical Inference

While recreational fishing surveys rely primarily on design-based methods, adopting some model-based inference could also yield benefits. Such inference might include Bayesian hierarchical models for finite populations and spatial-temporal models for fisheries management. These types of models could be helpful in dealing with missing data and handling differences in state and federal reporting procedures. Chapter 4 provides more information about how these models might be used (Recommendations 4-1 and 4-2).

Adoption of New Data-Science Technologies

Data-science technologies and computational frameworks are developing rapidly and providing ways of integrating additional sources of data with MRIP survey data. Chapter 3 provides examples of workflows that might be useful. These technologies are still developing, and it may be premature to formally adopt them within MRIP standards. Still, they are worthy of additional investigation as they have the potential to improve the timeliness, accuracy, and cost-efficiency of data (Conclusion 3-6). States and regional agencies will not have the same resources for developing such approaches. Thus, MRIP should provide guidance to them on how to use such technologies (Recommendation 4-3).

This page intentionally left blank.