An ad hoc committee under the auspices of the National Research Council will plan and organize a workshop to address statistical challenges in assessing and fostering the reproducibility of scientific results. To this end, the workshop will examine three issues from a statistical perspective: the extent of reproducibility, the causes of reproducibility failures, and potential remedies.
Specifically:
• What are appropriate metrics and study designs that can be used to quantify reproducibility of scientific results?
- Variability across studies is a well-known phenomenon and has given rise to the field of research synthesis and meta-analysis. How should this variability be assessed? What degree of variability would lead to concerns about lack of reproducibility?
• How can the choice of statistical methods for study design and analysis affect the reproducibility of a scientific result?
- How does routine statistical hypothesis testing with widely used thresholds for test significance affect the reproducibility of results? How do standard methods for study design and choice of sample size affect reproducibility?
• Are there analytical and infrastructural approaches that can enhance reproducibility, within disciplines and overall?
- Do we need new conceptual/theoretical frameworks for assessing the strength of evidence from a study? Do we need broad adoption of practices for making study protocols and study data available to the scientific community? How can this be achieved?
In addressing these three issues, the workshop will:
• Bring together representatives from different disciplines,
• Involve discussion of illustrative case studies, and
• Include some participants who are in positions to affect the incentive systems that could contribute to greater reproducibility of results.
The committee will plan and organize the workshop, select and invite speakers and discussants, and moderate the discussions. An individually-authored summary of the presentations and disucssions at the workshop will be prepared by a designated rapporteur in accordance with institutional guidelines.