Reproducibility and Replicability in Science

Reproducibility and Replicability in Science: Next Steps Symposium

September 24, 2019

8:30 AM - 6:00 PM (ET)

Past

Agenda

About

This symposium explored some of the main topics of the report and next steps toward improving research and how different audiences can gain confidence in scientific results. Four topis were explored through moderated discussions with panelists and the audience:

Actions and Responsibilities of Agencies and Scientific Leaders to Promote Confidence in Scientific Results
Replicability: What it Means to Replicate a Study
Reproducibility: Bridging the Gap between Funders and Providers of Needed Services for Computational Reproducibility
Moving Forward on Improving Science: Advice from Recent Academies Reports

Background Materials

Additional Information

Is it a Closed Session event?

Not Closed

Contact

Garret Tyson

(202) 334-1737
gtyson@nas.edu

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Reproducibility and Replicability in Science Public Release Webinar

May 7, 2019

1:00 PM - 2:00 PM (ET)

Past

About

The webinar includes an overview of the study process and discussion of the report’s conclusions, recommendations, and key messages.

Additional Information

Contact

Garret Tyson

(202) 334-1737
gtyson@nas.edu

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Reproducibility and Replicability in Science Meeting 4

May 9, 2018

12:00 PM - 1:00 PM (ET)

Past

Agenda

Download the Agenda

Materials

About

Perspectives on Reproducibility and Replication of Results in Climate Science

During the open session, the panel will explore the following questions:

• How has the awareness and understanding about reproducibility and replication in climate science evolved over recent years?
• Are there specific challenges regarding reproducibility that you have encountered or are aware of? Identify specific steps that are being taken, either by you or by others, to ameliorate these issues.
• Highlight historical and potential new approaches to reproducing and replicating climate science research using examples such as paleoclimate data to test models and estimate uncertainties.

Additional Information

Is it a Closed Session event?

Not Closed

Contact

Garret Tyson

(202) 334-1737
gtyson@nas.edu

Location

Keck Center

500 5th St NW

Washington DC 20001, USA

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Reproducibility and Replicability in Science Meeting 3

April 18 - 19, 2018

Past

Materials

About

The committee will hear about perspectives on scientific progress and irreproducibility, as well as perspectives on reproducibility from the physical and earth sciences, industry and industrial engineering, and economics.

Additional Information

Is it a Closed Session event?

Partially Closed

The following committee members were present at the closed sessions of the event

Harvey Fineberg
Juliana Freire
Lorena Barba
Edward Hall
Dianne Chong
Constantine Gatsonis
Gerald Gabrielse
David Allison
Victoria Stodden
Simine Vazire
Timothy Wilson
Wendy Wood
Dietram Scheufele
Thomas Jordan
David Donoho (via Zoom on Day 2)

The following topics were discussed in the closed sessions:

• the structure of the draft report and writing assignments
• the report’s main conclusions and recommendations
• review of the open session’s briefings

The following materials (written documents) were made available to the committee in the closed sessions:

No written materials were provided to the committee in closed session.

Date of posting of Closed Session Summary:

April 24, 2018

Contact

Garret Tyson

(202) 334-1737
gtyson@nas.edu

Location

Keck Center

500 5th St NW

Washington DC 20001, USA

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Reproducibility and Replicability in Science Meeting 2

February 22 - 23, 2018

Past

Agenda

Download the Agenda

About

Presentation on perspectives on reproducibility and replication from the American Economic Association

Additional Information

Is it a Closed Session event?

Partially Closed

The following committee members were present at the closed sessions of the event

Harvey Fineberg
Juliana Freire
Lorena Barba
Edward Hall
Dianne Chong
Constantine Gatsonis (via Zoom)
Gerald Gabrielse
David Allison
Victoria Stodden
Simine Vazire
Timothy Wilson
Wendy Wood
Dietram Scheufele
Thomas Jordan
David Donoho (via Zoom on Day 2)

The following topics were discussed in the closed sessions:

The committee:
reviewed its statement of task,
discussed possible topics for commissioned papers,
discussed initial report outlines, and
discussed possible topics for future meetings.

The following materials (written documents) were made available to the committee in the closed sessions:

No documents were made available to the committee in the closed sessions.

Date of posting of Closed Session Summary:

February 23, 2018

Contact

Garret Tyson

(202) 334-1737
gtyson@nas.edu

Location

Keck Center

500 5th St NW

Washington DC 20001, USA

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Meeting Closed

Reproducibility and Replicability in Science Zoom Meeting #5

December 17, 2018

9:30 AM - 12:00 PM (ET)

Past

About

International Perspectives on Reproducibility and Replication in Science and Engineering

During the open session, the panel will explore the following questions:

• Are there specific examples in your country/region where a lack of reproducibility and replicability in research results has led to doubt about reported results more broadly? Are reproducibility and replication of research results a global concern or is it a concern focused within specific countries?
• Are there particular scientific fields in which lack of reproducibility and replicability is more/less of a concern?
• Are there any concrete actions that organizations (e.g. funders, publishers, societies) in your country or region have taken to address concerns about reproducibility and replicability? What actions should they take?
• Should the research community work regionally and/or globally to address concerns about reproducibility and replicability? If so, what should be the priorities?

Additional Information

Is it a Closed Session event?

Institution Closed

The following committee members were present at the closed sessions of the event

Harvey Fineberg (Chair)
David Allison
Dianne Chong
Dietram Scheufele
Juliana Freire
Lorena Barba
Thomas Jordan
Timothy Wilson
Victoria Stodden
Wendy Wood

The following topics were discussed in the closed sessions:

The committee reviewed the latest draft of its report and discussed its main findings and recommendations.

The following materials (written documents) were made available to the committee in the closed sessions:

No documents were provided to the committee in closed session.

Contact

Lesley Webb

(202) 334-1663
lwebb274@gmail.com

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Meeting Closed

Reproducibility and Replicability in Science Zoom Meeting #4

November 11, 2018

3:00 PM - 5:00 PM (ET)

Past

About

Perspectives on Reproducibility and Replication of Results in Climate Science

During the open session, the panel will explore the following questions:

• How has the awareness and understanding about reproducibility and replication in climate science evolved over recent years?
• Are there specific challenges regarding reproducibility that you have encountered or are aware of? Identify specific steps that are being taken, either by you or by others, to ameliorate these issues.
• Highlight historical and potential new approaches to reproducing and replicating climate science research using examples such as paleoclimate data to test models and estimate uncertainties.

Additional Information

Is it a Closed Session event?

Institution Closed

The following committee members were present at the closed sessions of the event

Harvey Fineberg - Chair
Constantine Gatsonis
Dianne Chong
Dietram Scheufele
Ed Hall
Erin Forstag
Juliana Freire
Lorena Barba
Tim Wilson
Tom Jordan
Wendy Wood

The following topics were discussed in the closed sessions:

The committee reviewed the latest draft of its report and discussed its main findings and recommendations.

The following materials (written documents) were made available to the committee in the closed sessions:

No documents were provided to the committee in closed session.

Contact

Lesley Webb

(202) 334-1663
lwebb@nas.edu

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Closed

Reproducibility and Replicability in Science Zoom Meeting #3

October 11, 2018

10:00 AM - 12:00 PM (ET)

Past

Additional Information

Is it a Closed Session event?

Institution Closed

The following committee members were present at the closed sessions of the event

Diana Chong
David Allison
Michael Cohen
Juliana Freire
Harvey Fineberg
Dietram Scheufele
Tom Jordan
Timothy Wilson
Lorena Barba
Phone number
Erin Forstag
Gerald Gabrielse
Wendy Wood
Victoria Stodden

The following topics were discussed in the closed sessions:

The committee reviewed the latest draft of its report and discussed its main findings and recommendations.

The following materials (written documents) were made available to the committee in the closed sessions:

No documents were provided to the committee in closed session.

Date of posting of Closed Session Summary:

October 12, 2018

Contact

Garret Tyson

(202) 334-1737
gtyson@nas.edu

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Closed

Reproducibility and Replicability in Science Zoom Meeting #2

September 28, 2018

11:00 AM - 3:00 PM (ET)

Past

About

This meeting will be closed (a committee-only meeting) in its entirety.

Details will be posted after the meeting.

Additional Information

Is it a Closed Session event?

Institution Closed

The following committee members were present at the closed sessions of the event

Participant List:
Tom Jordan
Gatsonis
Lorena Barba
Scheufele
Simine
Tom Arrison
Ej Hall
Gabrielse
Erin Forstag
Victoria
TDW
Juliana Freire
Michael Cohen

The following topics were discussed in the closed sessions:

The committee reviewed the latest draft of its report and discussed its main findings and recommendations.

The following materials (written documents) were made available to the committee in the closed sessions:

No documents were provided to the committee in closed session.

Date of posting of Closed Session Summary:

October 04, 2018

Contact

Contact us

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Closed

Reproducibility and Replicability in Science Zoom Meeting #1

September 7, 2018

11:00 AM - 3:00 PM (ET)

Past

About

This meeting is closed to the committee-only details will be posted after the meeting.

Additional Information

Is it a Closed Session event?

Institution Closed

The following topics were discussed in the closed sessions:

The committee reviewed the latest draft of its report and discussed its main findings and recommendations.

The following materials (written documents) were made available to the committee in the closed sessions:

No documents were provided to the committee in closed session.

Date of posting of Closed Session Summary:

October 04, 2018

Contact

Contact us

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Closed

Reproducibility and Replicability in Science Meeting 6

July 24 - 25, 2018

Past

Additional Information

Is it a Closed Session event?

Institution Closed

The following committee members were present at the closed sessions of the event

All members were present physically with the exception of Wendy Wood, who participated via teleconference.

The following topics were discussed in the closed sessions:

The committee reviewed the latest draft of its report and discussed its main findings and recommendations.

The following materials (written documents) were made available to the committee in the closed sessions:

No documents were provided to the committee in closed session.

Date of posting of Closed Session Summary:

August 03, 2018

Contact

Garret Tyson

(202) 334-1737
gtyson@nas.edu

Location

J. Erik Jonsson Woods Hole Center

314 Quissett Avenue

Woods Hole MA 02543, USA

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Closed

Reproducibility and Replicability in Science Meeting 5

May 31 - June 1, 2018

Past

About

International Perspectives on Reproducibility and Replication in Science and Engineering

During the open session, the panel will explore the following questions:

• Are there specific examples in your country/region where a lack of reproducibility and replicability in research results has led to doubt about reported results more broadly? Are reproducibility and replication of research results a global concern or is it a concern focused within specific countries?
• Are there particular scientific fields in which lack of reproducibility and replicability is more/less of a concern?
• Are there any concrete actions that organizations (e.g. funders, publishers, societies) in your country or region have taken to address concerns about reproducibility and replicability? What actions should they take?
• Should the research community work regionally and/or globally to address concerns about reproducibility and replicability? If so, what should be the priorities?

VIEW THE VIDEO PLAYLIST

Additional Information

Is it a Closed Session event?

Institution Closed

The following committee members were present at the closed sessions of the event

Harvey Fineberg
Juliana Freire
Lorena Barba
Edward Hall
Dianne Chong
Constantine Gatsonis
Gerald Gabrielse
David Allison
Victoria Stodden
Simine Vazire
Timothy Wilson
Wendy Wood
Dietram Scheufele
Thomas Jordan
David Donoho (via Zoom on Day 1 and in the morning session of Day 2)

The following topics were discussed in the closed sessions:

• review of the open session’s briefings
• updates to the draft report and writing assignments
• the report’s main conclusions and recommendations
• plans for the committee’s writing meeting in July

The following materials (written documents) were made available to the committee in the closed sessions:

No written materials were provided to the committee.

Date of posting of Closed Session Summary:

June 12, 2018

Contact

Garret Tyson

(202) 334-1737
gtyson@nas.edu

Location

Keck Center

500 5th St NW

Washington DC 20001, USA

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Log In

Sign in to access your saved publications, downloads, and email preferences.

Former MyNAP users: You'll need to reset your password on your first login to MyAcademies. Click "Forgot password" below to receive a reset link via email. Having trouble? Visit our FAQ page to contact support.

Members of the National Academy of Sciences, National Academy of Engineering, or National Academy of Medicine should log in through their respective Academy portals.

Email (required)
Password (required)

Don't have an account?

Register

Email (required)
New password (required)
Retype password (required) Password must be at least 8 characters
I accept the terms of use
terms of use

Thank You

Thank you for creating a MyAcademies account!

Enjoy free access to thousands of National Academies' publications, a 10% discount off every purchase, and build your personal library.

Forgot Password

Enter the email address for your MyAcademies (formerly MyNAP) account to receive password reset instructions.

Reset Requested

We sent password reset instructions to your email . Follow the link in that email to create a new password. Didn't receive it? Check your spam folder or contact us for assistance.

Reset Password

New password (required)
Retype password (required) Password must be at least 8 characters

Reset Password

Your password has been reset.

Log In

Verify email

Verify Your Email Address

We sent a verification link to your email. Please check your inbox (and spam folder) and follow the link to verify your email address. If you did not receive the email, you can request a new verification link below

Subscribe

First Name
Last Name
Email (required)
I agree to receive emails from the National Academies (required)

We respect your privacy and will never share your information.

My Academies

Reproducibility and Replicability in Science

Dropdown items