During the course of their careers, most professional school students will be required to make decisions in which scientific and technical knowledge play a critical role. Yet many of these students will graduate without an adequate understanding of fundamental principles of scientific methodology or causal inference, and hence without a basis for understanding and assessing particular scientific claims. This failure to comprehend basic scientific research and reasoning may lead some policy professionals to be overly deferential to, or overly skeptical of, scientific claims, or leave them unable to distinguish claims based on scientific evidence from those claims based on political or other forms of interest. Such ignorance also may lead policy professionals to ignore or actively avoid engagement with scientific knowledge even when it is directly relevant to their work.
Students of law, public policy, medicine, journalism and business would benefit from early exposure to materials that elucidate the role that scientific information and scientific processes play in decision-making across diverse venues. In their professional lives, these individuals will invariably encounter scientific issues in legislation, regulation, standard-setting, litigation, and business decisions in subject matter areas ranging from energy and the environment to health and national security.
In light of the ever-increasing role that science plays in policy decisions, our committee, the Committee on Preparing the Next Generation of Policy Makers for Science-Based Decisions, an ad hoc committee under the auspices of the National Academies of Sciences, Engineering, and Medicine’s Committee on Science, Technology, and Law (CSTL), has overseen the development of educational modules to elucidate the role of science in decision-making for professional school students.
With these modules, we are not seeking to replicate graduate science programs in schools offering public policy training. It is neither feasible nor desirable to try to turn most professional school students into amateur scientists. It is possible, however, to present scientific principles in the context of public policy debates that involve issues that professional school students may encounter so they may become informed consumers of scientific research and information. We seek to empower students by providing access to materials that:
Expose them to basic scientific principles and approaches so that they understand the frameworks of scientific inference.
Enable them to understand the methodology of a scientific study and to ask questions that will inform an assessment of its evidentiary underpinnings and the validity of its claims.
Provide them with materials that elucidate the scientific process, illustrate how scientific communities operate, the qualifications of those who conduct scientific research, and provide insights into interactions between members of the scientific community and other communities.
Provide materials that illustrate how science is a community process that both builds on and challenges existing knowledge, and subjects empirical claims to evidence derived from observation and experiment.
Allow students to distinguish when there is genuine scientific consensus on an issue and when there are legitimate differences – and how to best make decisions on the spectrum from consensus to serious disagreement, and identify a path to a more informed understanding of relevant policy issues.
Background
This project is a product of CSTL’s longstanding investment in public policy formation, which includes looking at the uses and misuses of science in shaping public policy at the confluence of the scientific, engineering, medical, and legal arenas. It is a direct follow on to CSTL’s development, in collaboration with the Federal Judicial Center, of the third edition of the Reference Manual on Scientific Evidence. The Reference Manual assists judges in managing cases involving complex scientific and technical evidence by describing the basic tenets of key fields from which scientific evidence is typically derived.
The project began in January 2012 when CSTL assembled twenty individuals from foundations, the judiciary, academic law, public policy, and science for a roundtable discussion on the feasibility of a project to increase the:
science literacy and sophistication of law and public policy students by designing modules that could be used by faculty to create integrative, overview courses that addresses the nature of science, methods of scientific inquiry, standards of hypothesis testing and proof, and the uses and misuses of the natural and social sciences in both private and public decision-making.
The attendees agreed that such an effort would be valuable and, in the summer of 2012, a follow up meeting was held to refine the parameters of the project. This website and the materials contained herein are the end result of these discussions.
Professionals and educators across disciplines widely acknowledge that an understanding of fundamental scientific principles and techniques is crucial. In response to concerns about the science content in the current premedical and medical education curricula, for instance, the American Association of Medical Colleges (AAMC) and the Howard Hughes Medical Institute (HHMI), recently convened a group of scientists, physicians, and science educators from around the United States to determine the most important scientific competencies required of medical school graduates as they enter postgraduate training. The panel developed overarching principles which emphasized that “the acquisition, understanding, and application of scientific knowledge and scientific reasoning based on evidence” is essential to “the ability to evaluate competing claims in the medical literature.”1
In law, influential jurists, such as Justice Harlan Fiske Stone, have long highlighted the importance of interdisciplinary education. Stone’s ideas about the need to integrate economics, statistics, and social sciences into the study of law led him to lament the failure of law schools to pursue such interdisciplinary studies.2 As one scholar recently put it, “The typical J.D. curriculum provides no preparation in the skills needed to determine substantive facts beyond legal doctrine, nor in the scientific content and analytic methods necessary to assess and support factual conclusions.”3 Another scholar has noted that,
Many legal decisions cannot be made apart from their economic, social, historical, and political contexts, and are often dependent upon business, scientific, medical, psychological, and technological information. Nonlegal but legally relevant information is commonly used, among other reasons, to generate inferential support for factual premises, to support policy arguments, and to support or challenge legal rules.4
“If law is ever to fully incorporate science into its decision making,” writes law professor David L. Faigman, “law schools will need to provide the basic training.” “Law schools,” he continues, “are in the business of teaching critical thinking.” “At the beginning of the twenty-first century, it is no longer possible to be a critical thinking generalist if you cannot think critically about science.”5
Writing in the Harvard Business Review, Johan Roos, Dean and Managing Director of Jönköping International Business School, calls for a business school curriculum that is more STEM-driven. “We need,” he writes, “to increase business students’ knowledge about science, technology, engineering, and math.” “Given that so much innovation happens in STEM-oriented industries, business education needs to help students bridge the natural and social sciences.”6
The current project represents a continuation of the Academies’ commitment to the effective incorporation of the products of scientific knowledge in the development of public policy and a response to repeated calls for increased scientific literacy among non-scientists. A 1978 NRC report, Knowledge and Policy: The Uncertain Connection, examined, for instance, how the fruits of the scientific study of social issues might bear on the decisions of policy makers.7 More recently, in 2012, the Committee on the Use of Social Science Knowledge in Public Policy noted that:
policy education should equip its graduates to promote the use of science in policy-making settings…Policy students can be taught to appreciate policy making through policy argument or practical reasoning and to understand that the relevance of and weight given to science depends on the policy context. They can recognize the limits of the persuasive power of scientific reasoning, the substantial institutional barriers and cultural resistance to new scientific knowledge, and the role of moral and ethical beliefs.8
Our committee reaffirms the importance of the distinctive contribution of scientific knowledge to public policy debates and seeks to aid decision makers in government, business, medicine, and law in understanding the strengths and weaknesses of scientific information that seeks to inform policy debates. Scientific research offers a unique perspective on policy issues, derived from systemic methods intended to guard against self-deception and bias confirming a desired outcome. While other forms of knowledge also provide useful information, science provides a quantifiable method for assessing both the nature and the magnitude of problems and the consequences (both intended and unintended) of different forms of interventions. The committee hopes that the materials on this website will be used to strengthen the curriculum of professional schools and facilitate better understanding and collaboration amongst scientists conducting studies relevant to decision makers who must consider such findings, select a course of action, and guide its implementation.
The Modules
The committee developed a set of guidelines to facilitate the development of the educational modules and supplementary materials. It then asked prominent members of the legal and scientific communities to develop proposals for a range of topics where scientific information and understanding plays or played a critical role in decision making. The committee met with module authors to discuss their proposals and, on the basis of these discussions, contracts were awarded.
In the end, nine proposals were developed into modules. The modules are:
The Interpretation of DNA Evidence: A Case Study in Probabilities by David H. Kaye, The Pennsylvania State University
Models: Scientific Practice in Context by Elizabeth Fisher, Oxford University; Pasky Pascual, U.S. Environmental Protection Agency; and Wendy Wagner, University of Texas at Austin
Scientific Evidence of Factual Causation by Steve C. Gold, Rutgers University, Michael D. Green, Wake Forest University; and Joseph Sanders, University of Houston
Shale Gas Development by John D. Graham, John A. Rupp, and Adam V. Maltese, Indiana University
Drug-Induced Birth Defects: Exploring the Intersection of Regulation, Medicine, Science, and Law by Nathan A. Schachtman, Columbia University
Forensic Pattern Recognition Evidence by Simon A. Cole and Alyse Berthenal, University of California, Irvine and Jaclyn Seelagy, University of California, Los Angeles
Placing a Bet: A New Therapy for Parkinson’s Disease by Kevin W. Sharer, Harvard University
Translating Science into Policy: The Role of Decision Science by Paul Brest, Stanford University
Vaccines by Arturo Casadevall, Johns Hopkins University
The committee reviewed module drafts over the course of fifteen months. The committee had ultimate responsibility for approving the content of modules. For the duration of the project, the committee corresponded with module authors and suggested refinements and enhancements to submissions.
Materials were designed for use by faculty seeking to improve scientific understanding among legal, medical, policy, and business students, either as parts of courses or as a the basis for a course designed by individual faculty members. They have been structured to explicate core competencies9 in science and technology through illustrative case studies. While the modules are currently posted on this Academies’ website, the modules will ultimately be moved to a permanent external website that will, it is hoped, serve as a locus wherein additional educational materials may be discussed, developed, and housed.
By making exemplary educational resources readily accessible, faculty from various disciplines might easily design coursework to meet any or all of the goals stated above. The materials currently on the website have been designed to be sufficiently modular so as to allow faculty to incorporate specific components into a larger course – e.g., an evidence professor might make use of the “DNA” module within the basic evidence course, or an environmental policy professor might incorporate the “fracking” module into his or her seminar. Alternatively, multiple modules might be combined to form the basis for an integrated course on science, law, and public policy. The existence of a variety of materials in distinct topic areas provides faculty with a unique resource from which to draw materials to increase professional school students’ sophistication when confronting issues involving science.
The typical module is centered on a case or problem that elucidates some of these competencies and provides text, ancillary materials, and a guide for instructors. The modules may be used subject to the terms of the Creative Commons Attribution-NonCommercial 4.0 International Public License. As envisioned, the materials will serve as core reference material to guide developers of future modules.
This website housing the modules is not intended as a static collection of material. Rather, it is expected that the website and collection of materials will grow over time, and this is exactly the outcome that the committee ultimately seeks. We encourage professional school faculty and others to revise the educational materials on this website to fit their particular needs and to share this resource widely.
As policy makers confront new policy issues and as scientific and technical understanding continues to evolve, educational materials in professional schools must evolve to meet the needs of a changing scientific and policy landscape.
We wish to express our deep appreciation to the authors of the modules for their responsiveness to the suggestions of the committee and to the committee for their reviews of numerous module drafts and for their input on our project from conception to completion.
Finally, we would like to thank the staff of the committee; Steven Kendall, Charles Stotler, Karolina Konarzewska, and Anne-Marie Mazza.
Paul Brest
Former Dean and Professor Emeritus (active), Stanford Law School
and
Saul Perlmutter
Franklin W. and Karen Weber Dabby Professor, University of California, Berkeley and Senior Scientist, E. O. Lawrence Berkeley National Laboratory
Co-Chairs
Committee on Preparing the Next Generation of Policy Makers for Science-Based Decisions
Posted: June 2016
1American Association of Medical Colleges and the Howard Hughes Medical Institute, “Scientific Foundations for Future Physicians: Report of the AAMC-HHMI Committee,” 2009, p 5.
2See Miriam Galston, Activism and Restraint: The Evolution of Harlan Fiske Stone’s Judicial Philosophy, 70 TUL. L. REV. 137, 172-73 (1995).
3Carl N. Edwards, In Search of Legal Scholarship: Strategies for the Integration of Science into the Practice of Law, 8 S. CAL. INTERDISC. L.J. 1, 32, 1998.
4Steven M. Barkan, Response to Schanck, On the Need for Critical Law Librarianship, or Are We All Legal Realists Now? 82 LAW LIBR. J. 23, 34-35 (1990).
5David L. Faigman, The Law’s Scientific Revolution: Reflections and Ruminations on the Law’s Use of Experts in Year Seven of the Revolution, 57 WASH. & LEE L. REV. 661, 683-4 (2000).
6Johan Roos, “The Renaissance We Need in Business Education,” Harvard Business Review, July 2, 2014.
7National Research Council, Knowledge and Policy: The Uncertain Connection (Washington, DC: Printing and Publishing Office, National Academy of Sciences, 1978).
8National Research Council, Using Science as Evidence in Public Policy (Washington, DC: The National Academies Press, 2012).
9As it began to frame the project, the committee developed a list scientific issues/topics that the modules might address:
Scientific Methods
The probabilistic nature of most empirical knowledge
The world as it actually is vs. the world as we discern it
Type I vs. Type II errors
Confidence internals/margins of error
Linear regression, simple and multivariate
Correlation coefficient
Validity
Measurement errors, reliability
Testing hypotheses
Particular methods
Replication
Declining effects
Bayes’ Theorem
The difference between using statistics (1) to make generalizations about populations and (2) to judge the cause of a particular event or predict an outcome for a particular individual
The ideal of falsifiability and its limits
Different disciplinary approaches
Is there a fundamental difference between natural and social science methods?
Survey data
Sampling and public opinion polling
The growing role of administrative data (e.g., credit card use) as an alternative to sampling
The role of models (e.g., in climate, astrophysics)
Differences among approaches of various natural sciences (e.g., physics, astrophysics, chemistry, biology, clinical medicine)
Publication bias
Biases in perceiving, remembering, and analyzing information (judgment and decision making and social psychology)
Lay intuitions about probability and statistics
Confirmation bias
Anchoring, availability, representativeness, hindsight, etc.
Social influences
Scientific Culture and Communities
Nature of scientific consensus
The contingent nature of scientific standards and findings
Who sets standards?
Peer review and publication/circulation of data and results
Publication bias, replication
The tenure process
Paradigm-shifting visionaries vs. cranks
Role of learned societies
Role of NSF/NIH/other research agencies
Science Advisor/OSTP
Resolving scientific disputes
Scientific ethics:
Science and Society
What constitutes a scientific consensus and what are the implications for policy and business decisions?
Decision-making under uncertainty
Lay vs. expert, global vs. domain specific views of risk
Precautionary principle
Scientists’ involvement in policy and litigation
Issues of responsibility, expertise, and credibility
Science in the courts
Examples of Subject Areas Where Scientific Understanding is Critical to Making Informed Decisions
DNA
Genomics
Breast, prostate cancer and other screening
Forensic science
Water Quality
Electromagnetic radiation
Emerging technologies
Intellectual Property
Response to global disease
Scientific issues in remediation of environmental harm
Consumer genetic screening
Gene editing
Fetal testing and genetic screening for newborns
Genetically modified foods and health and safety
Advertising and labeling of food, drugs, etc vis-à-vis health and safety claims
Climate change
Health care
End of life
Reproductive rights
Stem cell research
Nuclear weapons proliferation and nuclear weapons security
Food safety
Environmental standards
Internet
Surveillance of potential terrorism through mathematical data mining of “private” information
Cybersecurity