5

Key Themes in This Report

This chapter sums up the key themes appearing in this report, organizing the material somewhat differently in terms of which recommendations can be adopted relatively quickly and which will require more development effort. This chapter is not intended to be comprehensive, instead focusing on key themes.

The development and implementation of data quality standards is important both for Marine Recreations Information Program (MRIP)sponsored surveys and for those surveys conducted by states and regional agencies. When states and regional agencies are involved, sharing a common set of standards becomes especially critical, helping to ensure that the data are compatible and can be aggregated meaningfully. This chapter highlights some of the key results of this consensus study of MRIP’s data quality standards.

MRIP has taken an important step by developing its data quality standards. The standards are also well designed. They are consistent with the standards used by federal statistical agencies and with other organizations. They are generally well accepted by those collecting and using the data. To the extent that this report is critical of the standards, it is largely to suggest ways of expanding beyond the current standards to cover a broader range of situations.

One aspect of the standards quickly drew complaints: the requirement that statistics with percent standard errors (PSEs) greater than 50 not be published. The complaints were along two lines: that the data are needed for fishery management purposes even if they are not fully reliable, and that access to such data varies, resulting in inequities in access. Handling data

with high standard errors is an issue on which federal statistical agencies and statisticians in general have been divided, perhaps particularly when there are clear policy implications based on using the data. Often agencies follow a strict rule of suppressing such data as unreliable, but sometimes the very same agencies make exceptions depending on the survey. The Federal Committee on Statistical Methodology instead focuses on fitness for use and allows for situations in which the data may be the best data available, even if the data are not fully reliable.

The panel acknowledges this diversity of approaches and favors transparency, that is, releasing such data along with clearly marked caveats about the reliability of the data. In the panel’s opinion, MRIP has handled this situation well by releasing the data along with the use of color coding to clearly mark those data that are less reliable.

Recommendation 3-4: In the interests of transparency and supporting fishery management programs, Marine Recreational Information Program should continue its current practice of publishing estimates with high percent standard errors while clearly identifying such estimates as having low reliability.

In the remainder of this chapter, the panel presents two categories of improvements that might be considered by MRIP: some that are relatively incremental in nature, so they can be adopted more quickly and possibly in stages, and others that will require more extensive study but also potentially offer great promise.

INCREMENTAL CHANGES TO THE STANDARDS

Timeliness

Currently, timeliness is not mentioned within the data quality standards. One desire often expressed by those involved in fishery management is for greater timeliness in releasing the data. People involved in fishery management often need current data while the fishing season is in progress, facing much more immediate needs for data than those engaged in annual planning. Concerns about timeliness are not unique to MRIP; timeliness is frequently an issue for federal data collections, partly due to logistics (with many actors involved in a wide range of activities) and partly due to quality control needs (with time required to process and edit the data). Thus, the panel recognizes that timeliness is not fully under MRIP’s control, and finding an appropriate balance between timeliness and high data quality may be difficult. Still, timeliness should be listed as an aspiration, and some of the topics discussed elsewhere in this report (e.g., the use of artificial intelligence [AI] for data processing, modeling, and non-probability sampling)

may be useful in improving timeliness. States’ cooperation may be enhanced if their needs for timeliness are considered in addition to MRIP’s own schedule.

Recommendation 2-4: Marine Recreational Information Program should add “timeliness” to its data quality standards to better meet the needs of those actively managing fisheries.

Consistency

The various surveys do not all collect data at the same level of detail, making it harder to aggregate the data and creating a need for calibration. There are sometimes good reasons for the variations: data collections vary depending on the mode of fishing used and on species characteristics (e.g., groundfish). Still, to the degree that MRIP can create standards requiring more consistent data, even if they vary by species and fishing mode, the data may become more reliable and require less processing time, perhaps accelerating the schedule. To the degree that agencies can become more consistent in the types of data and level of detail collected and reported across agencies, the need for calibration will be reduced. Improved consistency can also come through the adoption of standard operating procedures when appropriate. However, this latter topic was outside of the scope of the panel’s standards review.

Recommendation 2-5: Marine Recreational Information Program should evaluate how to promote greater consistency in how catches are enumerated.

Data Quality

While data reliability as measured by standard errors or coefficients of variance is certainly important, MRIP might consider broadening its requirements to consider total survey error, that is, non-sampling error as well as sampling error. One example is that measures of discards that are based on self-reports may underestimate the number of discards if people who were unsuccessful in their fishing are less likely to respond.

Burden and Confidentiality

Some requirements of MRIP, and in particular the requirement to provide both raw and edited data, are not standard practice and create the potential for the release of confidential information. The standards should be reviewed to prevent the release of confidential information.

Conclusion 3-4: The current documentation requirements may create risks for confidentiality and may present burdens on data providers that are contrary to standard practice.

Implementation

States sometimes expressed difficulty in meeting MRIP standards. It may be better to have partial compliance than to discourage states from seeking certification at all. For these reasons, there could be value in setting up a tiered system of certification, allowing states to participate at some level even if they cannot meet all of the requirements.

Recommendation 2-2: Marine Recreational Information Program should make its standards more manageable to states and agencies that lack statistical experts and should consider expanding the technical expertise on surveys and statistics it provides to state and regional agencies.

Recommendation 3-3: Marine Recreational Information Program should consider adopting tiered levels of certification reflecting different levels of data quality.

AREAS WORTHY OF EXPLORATION OFFERING ADVANCES FOR THE FUTURE

There have been important developments in model building and AI, and these have the prospect of providing great benefit to MRIP. These areas deserve to be studied to determine how they are developing and how they might be used. AI, in particular, is advancing so rapidly that its potential uses may greatly expand even within just a few years.

Model Building

Chapter 4 discusses the potential benefits that may arise from the use of model-based approaches, such as Bayesian hierarchical models for finite populations or spatial-temporal models that explicitly account for the interconnectedness of data points across locations and over time. Such model building could be used to better estimate the relationship between fishing effort and catch amounts, calibrate data when states differ in the data they collect, bridge data from other sources, to better understand population dynamics, evaluate the impacts of regulations and habitat changes, and develop improved survey and data collection designs. It will be important

to consider variability in the models and how to handle it—for example, by averaging across models or choosing those that seem most appropriate.

Recommendation 4-2: Marine Recreational Information Program should explore and, where appropriate, promote the use of model-based inference as a way of improving data quality and timeliness.

Adoption of New Data-Science Technologies

Recent developments in data-science and new technologies such as remote sensing and AI offer promise for improving data collection, processing, and management, and for integrating new types of data. These developments may lead to both higher quality data and greater timeliness.

Conclusion 4-2: New technologies are available that may be helpful in combining data from multiple sources and reconciling differences.

Recommendation 4-3: Given the limited resources available to many state and regional agencies, and also to avoid duplication of effort, Marine Recreational Information Program should provide guidance to the agencies on how to use technology to promote higher data quality.

PLANNING A TIMELINE

Since the current standards are already well designed, there is no urgency to make changes. Almost all the changes suggested here will require some level of study for the development phase, along with time for the states to adapt to the changes. MRIP might set a target of addressing the incremental changes in 1 to 3 years and the longer-range changes in 3 to 10 years. As part of the process, it will be important to allow sufficient time and provide vehicles for feedback from the states. The report specifically recommends this for changes to PSE reporting, but a consultative approach would be beneficial in other areas as well.

Recommendation 3-5: Because participating state agencies respond to their constituents, we suggest that Marine Recreational Information Program staff provide a forum to discuss the changes to percent standard error reporting that are recommended by this report and obtain feedback that is congruent with the needs of agencies and the advice from this report.

This page intentionally left blank.