Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop (2016)
Chapter: Front Matter
Statistical Challenges in Assessing
and Fostering the Reproducibility
of Scientific Results
Summary of a Workshop
Michelle Schwalbe, Rapporteur
Committee on Applied and Theoretical Statistics
Board on Mathematical Sciences and Their Applications
Division on Engineering and Physical Sciences
THE NATIONAL ACADEMIES PRESS
Washington, DC
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
This workshop was supported by Grant No. DMS-1351163 between the National Academies of Sciences and the National Science Foundation. Any opinions, findings, or conclusions expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project.
International Standard Book Number-13: 978-0-309-39202-0
International Standard Book Number-10: 0-309-39202-0
Digital Object Identifier: 10.17226/21915
Cover: The photo “Peas in a Pod” by Andrew_Writer is used with Creative Commons Attribution 2.0 Generic license (https://www.flickr.com/photos/dragontomato/3698617520/in/gallery-106928713@N04-72157638971870524/).
This report is available in limited quantities from:
Board on Mathematical Sciences and Their Applications
National Academies of Sciences, Engineering, and Medicine
500 Fifth Street NW
Washington, DC 20001
bmsa@nas.edu
http://www.nas.edu/bmsa
Additional copies of this workshop summary are available for sale from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu/.
Copyright 2016 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Ralph J. Cicerone is president.
The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. C. D. Mote, Jr., is president.
The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.
The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.
Learn more about the National Academies of Sciences, Engineering, and Medicine at www.national-academies.org.
This page intentionally left blank.
PLANNING COMMITTEE ON STATISTICAL CHALLENGES IN ASSESSING AND FOSTERING THE REPRODUCIBILITY OF SCIENTIFIC RESULTS
CONSTANTINE GATSONIS, Brown University, Co-Chair
GIOVANNI PARMIGIANI, Dana-Farber Cancer Institute, Co-Chair
STEPHEN FIENBERG, Carnegie Mellon University
STEVEN N. GOODMAN, Stanford University School of Medicine
JOHN H. HOLMES, University of Pennsylvania Perelman School of Medicine
ALAN F. KARR, RTI International
JELENA KOVAČEVIĆ, Carnegie Mellon University
XIHONG LIN, Harvard University
ROGER PENG, Johns Hopkins Bloomberg School of Public Health
VICTORIA STODDEN, University of Illinois, Urbana-Champaign
Staff
MICHELLE K. SCHWALBE, Senior Program Officer
RODNEY N. HOWARD, Administrative Assistant
LINDA CASOLA, Senior Program Assistant and Staff Editor
COMMITTEE ON APPLIED AND THEORETICAL STATISTICS
CONSTANTINE GATSONIS, Brown University, Chair
DEEPAK AGARWAL, LinkedIn
KATHERINE BENNETT ENSOR, Rice University
MONTSERRAT (MONTSE) FUENTES, North Carolina State University
ALFRED O. HERO III, University of Michigan
DAVID M. HIGDON, Virginia Bioinformatics Institute
ROBERT E. KASS, Carnegie Mellon University
JOHN LAFFERTY, University of Chicago
XIHONG LIN, Harvard University
JOSÉ M. F. MOURA, Carnegie Mellon University
SHARON-LISE T. NORMAND, Harvard University
GIOVANNI PARMIGIANI, Dana-Farber Cancer Institute
ADRIAN RAFTERY, University of Washington
LANCE WALLER, Emory University
EUGENE WONG, University of California, Berkeley
Staff
MICHELLE K. SCHWALBE, Director
RODNEY N. HOWARD, Administrative Assistant
LINDA CASOLA, Senior Program Assistant
BOARD ON MATHEMATICAL SCIENCES AND THEIR APPLICATIONS
DONALD SAARI, University of California, Irvine, Chair
DOUGLAS N. ARNOLD, University of Minnesota
JOHN B. BELL, E. O. Lawrence Berkeley National Laboratory
VICKI M. BIER, University of Wisconsin, Madison
JOHN R. BIRGE, University of Chicago
L. ANTHONY COX, JR., Cox Associates, Inc.
MARK L. GREEN, University of California, Los Angeles
BRYNA KRA, Northwestern University
JOSEPH A. LANGSAM, Morgan Stanley (retired)
ANDREW W. LO, Massachusetts Institute of Technology
DAVID MAIER, Portland State University
WILLIAM A. MASSEY, Princeton University
JUAN C. MEZA, University of California, Merced
CLAUDIA NEUHAUSER, University of Minnesota
FRED S. ROBERTS, Rutgers University
GUILLERMO R. SAPIRO, Duke University
CARL P. SIMON, University of Michigan
KATEPALLI SREENIVASAN, New York University
ELIZABETH A. THOMPSON, University of Washington
Staff
SCOTT T. WEIDMAN, Director
NEAL GLASSMAN, Senior Program Officer
MICHELLE K. SCHWALBE, Senior Program Officer
RODNEY N. HOWARD, Administrative Assistant
BETH DOLAN, Financial Associate
This page intentionally left blank.
Acknowledgment of Reviewers
This workshop summary has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published workshop summary as sound as possible and to ensure that the workshop summary meets institutional standards for objectivity, evidence, and responsiveness to the project’s charge. The review comments and draft manuscript remain confidential to protect the integrity of the study process. We wish to thank the following individuals for their review of this workshop summary:
C. Glenn Begley, TetraLogic Pharmaceuticals,
Joel Greenhouse, Carnegie Mellon University,
Joelle Lomax, Science Exchange, and
Randy Schekman, University of California, Berkeley.
Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the views presented at the workshop, nor did they see the final draft of the workshop summary before its release. The review of this workshop summary was overseen by Lawrence D. Brown, University of Pennsylvania, who was responsible for making certain that an independent examination of this workshop summary was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this summary rests entirely with the author and the institution.
This page intentionally left blank.
Contents
Opening Remarks from the Workshop Co-Chairs
Perspectives from Stakeholders
Overview of the Statistical Challenges of Reproducibility
3 CONCEPTUALIZING, MEASURING, AND STUDYING REPRODUCIBILITY
Definitions and Measures of Reproducibility
Reproducibility and Statistical Significance
Assessment of Factors Affecting Reproducibility
Reproducibility from the Informatics Perspective
4 THE WAY FORWARD: USING STATISTICS TO IMPROVE REPRODUCIBILITY
Open Problems, Needs, and Opportunities for Methodologic Research
Reporting Scientific Results and Sharing Scientific Study Data
The Way Forward from the Data Sciences Perspective: Research
