3

The Seven Standards: A Review

This chapter reviews the seven Marine Recreational Information Program (MRIP) data quality standards one by one. Each standard is first presented verbatim (see Boxes 3-1–3-7), followed by a discussion.

STANDARD 1: SURVEY CONCEPTS AND JUSTIFICATION

The panel accepts this data standard as an appropriate starting point for evaluating any proposed survey.

The panel notes the need for greater clarity to the first bullet, which might be read as saying the survey goal is to meet the key estimate and precision requirements. Though the production of reliable statistical estimates is certainly an important goal, a well-designed survey plan will also discuss what data are to be collected and how the data are intended to be used. A well-designed plan would include details on which species are the priority to monitor and what fishing seasons apply; however, MRIP might consider adding more detail than appears in Standard 7 to promote greater consistency in the measures used across states (e.g., how to measure the finfish catch, perhaps with different instructions for different species).

Following the American Association for Public Opinion Research (AAPOR) Best Practices for Survey Research cited above, MRIP’s survey concepts should emphasize that survey practitioners should consider the specific design of the study during the planning phase itself. In well-designed surveys, the plan would specify the mode of data collection (i.e., web, telephone, in-person, mail, or text, alone or in combination). Survey practitioners who wish to include non-probability sources similarly would define

their sources, including opt-in or volunteer panels, angler mobile apps, citizen science, and volunteered information. Another key part of the survey design process is to plan for appropriate weighting and post-adjustments regardless of whether probability samples, non-probability samples, or a blend of both are used. These are all key elements of the survey design process; model-based approaches are discussed separately in Chapter 4.

Data precision also depends on the tools that are used for measurement. Sometimes survey personnel are present at the landing sites to perform measurements themselves, but sometimes the surveys are dependent on anglers’ self-reports. This is especially true for discards. Tools such as phone apps can be used to help anglers provide more precise information.

The panel applauds the parallel consideration of survey objective and intended users. The panel believes that the joint consideration of both the objectives of the survey and the application of the outputs of the survey by the intended users will promote the development of surveys that have broadest impact and acceptance.

Conclusion 3-1: The current Marine Recreational Information Program standards are highly focused on survey research as the appropriate tool for collecting statistics, without considering alternative methodologies including, but not limited to, catch and landing records, electronic monitoring and vessel monitoring systems, and alternative modeling approaches.

Conclusion 3-2: The use of technology such as fishing measurement phone apps, which can be used to estimate biological characteristics of fish both caught and released and caught and retained, might improve both timeliness and accuracy of Marine Recreational Information Program estimates.

STANDARD 2: SURVEY DESIGN

Recognizing that survey design is a critical and central element to data standards, the panel finds that this standard is well written in terms of what it covers but also finds its coverage to be overly narrow.

The panel concludes that the data standard is narrow in its strong, direct, and limited focus on surveys as the principal or sole method of meeting the need that justifies the sampling. The panel suggests the standard be broadened to include both probability and non-probability sampling. The existing data standard is primarily focused on probability sampling, though it does allow for non-probability sampling, a newly developing area that may deserve more attention. Researchers may wish to combine or blend non-probability with probability data to increase the overall sample size or because of the need for convenience-based designs. When a non-probability sample is used and weights are calculated, an important step is to calibrate the weights to population totals from a reliable probability sample.¹

One key aspect of statistical design is the conceptual development and testing of whatever data are to be collected and of how those data can be collected in a way that will provide the most accurate and useful responses. Standard 2 implicitly addresses such issues in section 2.4 (evaluation) in the discussion of non-sampling error, but the objective is focused more on how to deal with or measure non-sampling error than on how to prevent it. There should be greater emphasis in the survey design phase on how to prevent or minimize non-sampling error, using such means as pretests and cognitive interviews. Standard 1.2, which requires compliance with the

___________________

¹ Bayesian modeling is discussed in greater detail in Chapter 4. Additional sources on nonprobability sampling include Liu et al. (2021, 2025), Salvatore et al. (2024), Wisniowski et al. (2020), and Yang et al. (2020).

Paperwork Reduction Act, will help in getting survey designers to consider non-sampling error as part of the design process, but it would be good to discuss this more explicitly.

Researchers should also be aware of and test for the potential for bias, particularly if the convenience-based design systematically excludes or underrepresents some types of people. For example, bias may occur if anglers who are successful in their catches are more likely to respond than anglers who are unsuccessful. Re-weighting to national totals does not necessarily correct for such bias. Calibration is a current area of research among survey practitioners with methods including LASSO (least absolute shrinkage and selection operator), propensity weighting, and small-area modeling described in Chen et al. (2018), Ganesh et al. (2017), Golini and Righi (2024), Valliant (2020), and Yang et al. (2020).

Conclusion 3-3: Standard 2 has a relatively narrow focus, concentrating on survey research as the methodological tool and primarily on probability sampling within survey research. Potential value may be gained by considering non-probability sampling.

STANDARD 3: DATA QUALITY

The panel finds this data standard to be clearly written. The requirement in Standard 3.1 to include both edited and unedited values in survey data sets goes beyond industry-standard approaches for constructing public-use data sets that contain edited data. While documenting the methods and procedures used to edit the data is considered a best practice, it is not

common practice to provide both the edited data and the original raw data when the data have been edited. Including flag variables to indicate which cases were subjected to specific types of edits, such as top-coding and imputation, is generally viewed as a good practice. However, including the unedited data may be problematic in that doing so might directly conflict with standards on confidentiality. That is, if enough data elements are available, then in combination they may make it possible to identify a respondent; data edits (particularly rounding and the use of categories) are sometimes applied as a measure to mitigate against disclosure risk.

Conclusion 3-4: The current documentation requirements may create risks for confidentiality and may present burdens on data providers that are contrary to standard practice.

The panel encourages MRIP staff to consider a total survey error (TSE) approach. The TSE framework is a concept in the survey research field that is useful when considering errors and their sources (Biemer, 2010). Errors are defined as being due to sampling error or non-sampling error, with the latter category further broken down into four types: coverage, nonresponse, measurement, and processing error (Biemer, 2010). As an example, problems with bias in who provides data on fishing discards may be more important than the sampling error associated with these statistics.

For research practitioners seeking to follow Standards 2.4 and 3.1, it may be helpful to describe sampling and non-sampling errors and their sources following a TSE framework. Specifically, item 2.4 on “Evaluation” within Standard 2 would relate to sampling error, and item 3.1 on “Processing, Editing, and Quality Control” within Standard 3 concerns non-sampling errors. Recent literature on this topic that would aid in the adoption of a TSE framework includes Biemer et al. (2017) and Amaya et al. (2020). In addition, Biemer and Lyberg (2003) provides more detail on quality control and evaluation across the survey lifecycle, exploring questionnaire design, interviewer effects, mode effects, and data processing while considering cost and timeliness.

Conclusion 3-5: The standards do not give sufficient consideration to reporting error and other non-sampling error, particularly with regard to discards.

MRIP was designed to provide the nation with the magnitude of harvests from recreational anglers based on the technologies available at the time. Newer technologies have the potential to affect both the collection and the analysis of data. Quality assurance and quality control constitute one area where artificial intelligence (AI) is likely to help reduce the burden on researchers going forward. For example, machine learning, natural

language processing, and deep learning have been applied to such tasks as data cleaning, de-duplication, data transformation, and data processing. Such approaches are expected, ultimately, to improve the accuracy and precision of data sets generated as part of MRIP. It is expected that AI will provide faster and more accurate methods of quality assurance in the coming years, potentially reducing costs while improving data quality. AI is discussed in greater detail in Chapter 4.

Conclusion 3-6: Requiring that data submitted to Marine Recreational Information Program include the original raw data as well as edited values may present risks to protecting confidentiality.

Conclusion 3-7: Artificial intelligence may be useful in supporting data quality checks and in helping such checks to be performed in a timely manner.

Recommendation 3-1: Marine Recreational Information Program should drive innovation by evaluating and encouraging the adoption of new technologies that enhance data precision and accelerate the release of information.

STANDARD 4: TRANSITION PLANNING

The panel supports the intent of this data standard. There has been a history of concern on the part of managers when MRIP has changed survey elements. Catch estimates substantially changed when MRIP replaced Marine Recreational Fisheries Statistics Survey as a result of the adoption of probability-based site allocation in the Access Point Angler Intercept Survey (APAIS). For the Fisheries Effort Survey, similar concerns were expressed within MRIP when the current mail survey replaced a telephone survey, while more recent concerns relate to the issue of question sequence.

National Marine Fisheries Service has conducted calibration studies for each of these types of transitions to measure how changes to the questionnaires or procedures have resulted in changes to the data, although some of these have been limited because of funding restrictions.

Multiple years of calibration may be beneficial if changes are substantive. Another major transition has been from MRIP’s oversight of APAIS to state involvement in the collection of survey data. It is important to know whether apparent changes in data reflect real changes in fishing practices or fishing stocks or are a result of changes in methodology. Similarly, it is possible that changes in methodology could hide or minimize actual changes in practices or stocks.

Given these concerns, Standard 4 is well justified. With any transition, consideration should be given to the potential for the change to affect estimates, affecting both current measures and finishing management procedures as well as measures of change over time; if the change has the potential to affect estimates, transition planning should include detailed descriptions of efforts to provide a data series to calibrate the impact of moving from the previous methodology to the new methodology. It can be difficult to anticipate what the impacts of changes might be; as for any change in survey design, it is wise to engage in at least some testing of the planned changes.

Conclusion 3-8: Anticipating and measuring the impact of changes to survey design, data collection, and data processing is critical to being able to accurately measure change over time.

STANDARD 5: REVIEW PROCEDURES

The above standard is comprehensive and is consistent with those of other organizations, including the AAPOR transparency initiative. That initiative goes further by having an official process where organizations are certified and undergo re-certification every two years.²

The panel recommends that MRIP survey practitioners follow the official AAPOR standard and calculations to estimate response rates to avoid ambiguity.³ The official AAPOR Standard Definitions (AAPOR 2023) detail how case dispositions are assigned and how the various rates listed as part of Standard 5 are calculated.⁴ AAPOR has also published web tools to as-

___________________

² https://aapor.org/standards-and-ethics/transparency-initiative/#where-can-i-get-more-info

³ https://aapor.org/standards-and-ethics/standard-definitions/

⁴ https://aapor.org/wp-content/uploads/2024/03/Standards-Definitions-10th-edition.pdf

sist in the calculation of official response rates,⁵ which may be of interest to newer users.

Recommendation 3-2: Marine Recreational Information Program should adopt American Association for Public Opinion Research standards for computing survey response rates.

___________________

⁵ https://aapor.org/response-rates/

Ideally, each survey program would meet each of these standards. However, in recognition of the challenges faced by some programs, including resource limitations, a tiered certification system could be considered. Under such a structure, programs meeting some but not all standards could be certified but with different levels indicating the level of adherence to the standards. This approach is used in some international educational assessment programs (see, for example, the Organisation for Economic Cooperation and Development’s Programme for International Student Assessments [PISA] program, which uses annotations in its reporting to indicate countries/economies that did not meet particular technical standards.⁶)

To implement a tiered system, consideration must be given as to the minimum criteria for each level (tier). For example, a program may be assigned one tier of certification if most (but not all) standards are met (or perhaps if key standards are met), a higher tier if all standards are met except full comparability of the data definitions with the federal survey, and a still higher tier if the definitions are designed to provide comparable data to the federal MRIP surveys. The exact specifications required for each tier would depend on MRIP’s priorities along with considerations of which requirements present the greatest difficulties for those seeking certification. Such a tiered system can help states to be involved in MRIP, allow MRIP data to be more complete, and provide information to users about the quality of the data.

Conclusion 3-9: Providing multiple tiers of certification or participation is one way that organizations have expanded access to participants meeting the quality standards to varying degrees.

Recommendation 3-3: Marine Recreational Information Program should consider adopting tiered levels of certification reflecting different levels of data quality.

STANDARD 6: PROCESS IMPROVEMENT

The panel supports this data standard. Over time, process improvement may occur for a variety of reasons. For example, some data needs might not currently be met satisfactorily, developments in technology or statistical processes may allow new opportunities for improvement in data collection and reporting, and changes in the environment may create the need for changes in process (as in MRIP’s earlier change from phone-based to address-based sampling). As Standard 6.2 notes, sometimes there may be a need to make changes simply to minimize or resolve a problem that has appeared. In a time of limited resources, how process improvements

___________________

⁶ https://www.oecd.org/en/about/programmes/pisa.html

are targeted and prioritized should be driven by consideration of the survey objectives (Standard 1). All changes should be documented, and the implications for data quality and comparability should be considered.

Conclusion 3-10: While it is desirable to pursue improvements in process, it is important to determine the impact of those improvements on survey estimates and prioritize development and implementation of improvements accordingly.

STANDARD 7: ACCESS AND INFORMATION MANAGEMENT

This is the most detailed and specific of the seven data standards. It has also been the most contentious of the draft data standards. As discussed in Chapter 1, MRIP standard that drew the greatest number of complaints from data users is the requirement to suppress estimates for which the percent standard errors (PSEs) exceeded 50 percent. The complaints had three main components: (a) even if unreliable, the estimates are often the only ones available and are needed for fishery management; (b) some users had access to the suppressed statistics, leading to inequities in data access; and (c) transparency is needed in order to promote trust in the data system.

The panel reviewed the issue from several perspectives:

• The original rule was based on the observed practices of federal statistical agencies and other programs (Richard Cody commented that most used a cutoff of 30 percent, with survey programs varying in how estimates are presented), but MRIP liberalized the cutoff somewhat to 50 percent.⁷ The Census Bureau itself sometimes uses a 50 percent threshold: for example, the American Community Survey uses a 50 percent threshold for PSEs (Census Bureau 2024), though it uses other rules for data suppression as well (e.g., the size of the population and whether, in a detailed table, more than half of the coefficients have a Coefficient of Variation [CV] greater than 0.61).
• Other federal statistical agencies often have a rule for suppressing data, perhaps more commonly based on CVs than on PSEs, but this practice is not universal, with some instead emphasizing transparency.
  • - The Bureau of Justice Statistics states that “BJS should ensure that all statistics it collects are disseminated to the public as widely as possible, using all appropriate media” (Guideline 6.1) while also stating, “BJS statistical products must provide sufficient information to permit the reader to evaluate the strengths and weaknesses of the data” (Guideline 4.2).⁸ However, Indicators of Crime and Safety, which is jointly published by BJS and the National Center for Education Statistics, suppresses statistics with CVs of 50 percent or greater (Irwin et al., 2024).
  • - The U.S. Department of Agriculture’s National Agricultural Statistics Service, for its 2019 Organic Survey, flagged statistics with a CV of 30 percent or higher, but did not suppress them, including for CVs greater than 50 percent (U.S. Department of Agriculture, n.d.).
  • - The Energy Information Administration, in its EIA Residential Energy Consumption Survey, does not publish estimates when the Relative Standard Error is greater than 50 percent but issues a complete microdata file that would allow such statistics to be computed, with a recommendation that the same guidelines be used.⁹
• Other federal agencies that publish statistical data but are not formally classified as federal statistical agencies (MRIP falls in this

___________________

⁷ Richard Cody, presentation to the panel on April 17, 2025.

⁸ https://bjs.ojp.gov/bjs-data-quality-guidelines/overview#21-0

⁹ https://www.eia.gov/consumption/residential/data/2020/pdf/microdata-guide.pdf

• category) sometimes choose to emphasize transparency, publishing all statistics while also flagging those data that are less reliable.
  • - The National Park Service, in its release of annual visitation statistics, labels current-year data as preliminary and notes other limitations (e.g., that parks sometimes rely on estimated data) but does not suppress data.¹⁰ Thus, it shows a preference for transparency over reliability, though its data issues were other than high PSEs.
• Robert Sivinski, representing the Federal Committee on Statistical Methodology, noted that best practices often are qualitative in nature (rather than specifying a numeric threshold) and are focused on transparency, putting the burden of assessing fitness for purpose on the user.¹¹
• Sometimes no alternative data are available, and those involved in fishery management need the estimates even if they are unreliable. Certain types of data, such as those for rare species, for short fishing seasons, and for discards, are especially difficult to collect, and the statistics are likely to be problematic.¹² Yet the data are still needed.
• David Marker (Marker Consulting, LLC) stated that no single set of rules is best in all situations, and there may be times when having bad data may be better than having no data. Marker noted that in the 1990s, many states’ estimates of the percentage of companies offering health insurance coverage to their employees had CVs greater than 50 percent, but there was a determination to release the estimates with a caveat about the low reliability.¹³ Marker thought the color coding scheme used by MRIP was effective in communicating concerns about the reliability of MRIP estimates.

Weighing these various factors, the panel felt that priority should be given to transparency, providing the statistics along with sufficient information for data users to evaluate the quality of the data. Suppressing the data would deprive some data users of needed data, while publishing the data with caveats about the data reliability allows data users to weigh what importance to give to the data.

___________________

¹⁰ https://www.nps.gov/subjects/socialscience/visitor-use-statistics-dashboard.htm

¹¹ Robert Sivinski, presentation to the panel on June 26, 2025.

¹² John Frazer, presentation to the panel on June 26, 2025; John Walter, presentation to the panel on May 23, 2025.

¹³ David Marker, presentation to the panel on June 26, 2025.

Conclusion 3-11: Transparency is important when data quality is at issue, allowing users to judge data fitness.

Conclusion 3-12: Having some data, even of poor quality, is sometimes better than having no data.

Conclusion 3-13: Marine Recreational Information Program’s current approach of using color coding to identify less reliable estimates is an effective means of communicating the need for user care.

Recommendation 3-4: In the interests of transparency and supporting fishery management programs, Marine Recreational Information Program should continue its current practice of publishing estimates with high percent standard errors while clearly identifying such estimates as having low reliability.

Recommendation 3-5: Because participating state agencies respond to their constituents, we suggest that Marine Recreational Information Program staff provide a forum to discuss the changes to percent standard error reporting recommended by this report and obtain feedback that is congruent with the needs of agencies and the advice from this report.

Since Standard 7 describes what data should be delivered to MRIP, its implementation also depends on other issues discussed elsewhere in this report. The encouragement to seek greater consistency in definitions and in what data are collected (Conclusion 2-6 and Recommendation 2-1) would affect the data deliveries described in Standard 7. Similarly, the panel’s concerns about protecting confidentiality (Conclusions 3-4 and 3-6) and for reporting on error (Conclusion 3-5) would affect data deliveries. The use of model building (Recommendation 4-2) and AI (Recommendation 4-3) could enhance data quality and consistency. Also, while Standard 7 concerns how data should be submitted to MRIP, one might also consider how data should be provided by MRIP; besides offering tables of information, MRIP might also consider offering visual or interactive presentations of data, making tables readily downloadable, archiving the data, and providing metadata.