The National Academies of Sciences, Engineering, and Medicine will empanel an ad hoc committee of approximately 13 – 14 experts who will assess research and data reproducibility and replicability issues, with a focus on topics that cross disciplines.

The committee will:

1. provide definitions of "reproducibility" and "replication" accounting for the diversity of fields in science and engineering,

2. assess what is known and, if necessary, identify areas that may need more information to ascertain the extent of the issues of replication and reproducibility in scientific and engineering research,

3. consider if the lack of replication and reproducibility impacts the overall health of science and engineering as well as the public’s perception of these fields,

4. review current activities to improve reproducibility and replication,

5. examine (a) factors that may affect reproducibility or replication including incentives, roles and responsibilities within the scientific enterprise, methodology and experimental design, and intentional manipulation; (b) as well as studies of conditions or phenomena that are difficult to replicate or reproduce,

6. consider a range of scientific methodologies as they explore research and data reproducibility and replicability issues, and

7. draw conclusions and make recommendations for improving rigor and transparency in scientific and engineering research and will identify and highlight compelling examples of good practices.

While the committee may consider what can be learned from past and ongoing efforts to improve reproducibility and replication in biomedical and clinical research, the recommendations in the report will focus on research in the areas of science, engineering and learning that fall within the scope of the National Science Foundation.

In addressing the tasking above, the committee may consider the following questions:

· Using definitions of “reproducibility” and “replicability” endorsed by the committee, explore what it means to successfully reproduce/replicate in different fields? Which issues (e.g. perhaps pressures to publish, inadequate training) are common across all or most fields when there are failures to replicate results?

· What is the extent of the absence of reproducibility and replication? Is there a framework that outlines the various reasons for lack of reproducibility and replication of a study?

· What strategies have scientists employed other than replication/reproducing findings to gain confidence in scientific findings (e.g., in situations where replication/reproducing is not possible, such as studies of ephemeral phenomena), and what are the advantages/shortcoming of those approaches?

· What cost-effective reforms could be applied? Where would they be best applied? What would their anticipated impact be?

Early in the process and throughout the study, scientific and engineering societies, communication experts, scientific tool developers, and other stakeholders will be engaged in the work of the committee as part of the data gathering process. These same stakeholder groups will be tapped at the end of the study in the planned release event, to ensure a wide distribution of the report.