On November 19–20, 2002, fifty-one invited participants from the fields of science, technology, engineering, and mathematics (STEM), postsecondary education, and education policy, along with National Research Council (NRC) staff and other interested parties attended a two-day workshop in Washington, D.C. at the National Academies. Participants and presenters were asked to explore three related issues: (1) how appropriate measures of undergraduate learning in STEM courses might be developed; (2) how such measures might be organized into a framework of criteria and benchmarks to assess instruction; and (3) how departments and institutions of higher learning might use such a framework to assess their STEM programs and to promote ongoing improvements. Participants covered a diverse set of topics and questions in addressing these issues. This document is not intended as a consensus report of the participants, and the steering committee provides no specific recommendations. Rather, the workshop was intended as an information-gathering activity by the Committee on Undergraduate Science Education (CUSE).

Several overriding concerns are highlighted here because workshop participants raised them numerous times. This Epilogue focuses especially on those topics and questions that call for further investigation, because addressing them more fully could have an important influence on improving education in all of the science disciplines and at all levels. The concerns addressed during the workshop were as follows.

In response to the first framing issue about developing measures of student

learning, several speakers and discussants including Brian Reiser, Northwestern University, and Gloria Rogers, Rose-Hulman Institute of Technology, argued that in preparing an effective science course, faculty must identify explicit, measurable learning objectives or outcomes (defined as what students need to know and be able to do by the end of each unit of instruction). A critical question was whether learning outcomes should be limited to a list of content terms, or—as proposed by one of the workshop speakers—should they consist of a framework of facts, central concepts, reasoning skills, and competencies such as the skills needed to think critically, an understanding of what constitutes evidence, and the ability to design a simple experiment?

With reference to issue 2, Paula Heron, University of Washington, described evidence that students’ preconceptions may be resistant to change by traditional didactic instruction such as lecturing. She demonstrated that students come to any topic with prior beliefs and conceptions that are often incomplete or erroneous, and that require carefully designed, specific measures to correct. Participants repeatedly acknowledged that judging an instructor’s knowledge and skill in applying such measures should be among the criteria for assessing instruction and for evaluating the extent to which instructors have at their command a variety of teaching strategies, in addition to lecturing, that are able to elicit a correct and deeper understanding of the subject on the part of students. According to the evidence reviewed, lecturing promotes memorization of factual information while more effective instruction that helps students gain functional knowledge requires teaching methods that assist them in explicitly reconciling their preconceptions with new information.

Further in response to issue 2, participants discussed the need for better assessment tools for evaluating course design and effective instruction. Anton Lawson, Arizona State University, emphasized results showing that the Reformed Teaching Observation Protocol (RTOP) could serve as a useful instrument for judging some aspects of teaching, and recommended that this tool might serve as a model for an

expanded instrument for that purpose. An evaluation instrument such as RTOP could be improved, suggested several participants, by including criteria for evaluating instruction that focus more on its success in eliciting defined learning outcomes among students.

Participants recognized that research on assessing and delivering undergraduate instruction is urgently needed, especially studies of how to improve teaching of large classes. Investigations in every subject area are necessary to identify students’ difficulties with specific concepts in each discipline. Research is also needed to find effective combinations or sequences of problem-solving, inquiry-based and didactic instructional practices to achieve student understanding of both basic concepts and the processes of scientific thinking. Arguments were presented by participants that much of that research could be done by scientists who are thoroughly grounded in the discipline, or by collaboration between such scientists and colleagues in education research.

Seeking models from which to extract traits to include in a set of criteria and benchmarks for evaluating instructional programs, workshop participants drew upon their own knowledge and experience to identify programs and curricula that are reputed to represent models of various forms of effective instruction. In the absence of a systematic national survey, those cited repeatedly were Biology Guided Inquiry Learning Environments (BGuILE), New Traditions, Peer-Led Team Learning (PLTL), Physics by Inquiry, Studio Physics, Workshop Physics, Problem-based Learning (PBL), and Case-Study Teaching. These programs are all characterized by some or all of the following traits: they have as their major aim to elicit specific factual, conceptual and cognitive learning outcomes on the part of students; they recognize that students have diverse learning styles and that they learn in different ways under different circumstances; they provide experiences for students to develop functional understanding of science concepts by using knowledge in investigations and problem-solving exercises and by making interdisciplinary connections; they promote students’ ability to work cooperatively, to communicate orally and in writing, and to develop independent learning skills; and they address the relevance of both science content and the processes of science to students’ lives.

Faculty who become instructional innovators and effective teachers share certain characteristics such as their expressions of equal respect toward academic staff, graduate students, and undergraduates, their command of a variety of instructional strategies that promote students’ conceptual understanding, and their ability to apply knowledge in new situations. These findings of an ethnographic study by Susan Millar, University of Wisconsin, served as a central point of discussion for the participants with reference to issue 3.

One element of issue 3 that participants focused on was whether opportunities and incentives for faculty to become familiar with different modes of instruction are sufficient to provoke needed changes in teaching? In responding to this question, several speakers and participants including Robert Zemsky, University of Pennsylvania, Herb Levitan, National Science Foundation, and Jack Wilson, UMassOnline, addressed the need to change the entire culture of higher education by a “top-down and bottom-up” approach. In the present structure of most institutions of higher learning, especially in research-intensive universities, incentives for faculty to learn new teaching methods are few.

Some of the strategies by which presidents, deans, and department chairs might encourage such cultural change included publicly announcing a fund earmarked for the support of faculty efforts to develop new courses; rewarding faculty efforts to improve instruction by allotting release time, summer stipends, or sabbatical leave; modifying promotion and tenure policies in ways that motivate faculty to spend time and effort on developing new teaching methods or redesigning courses to be more learner centered; providing instruction and mentoring for graduate students, postdoctoral fellows, and faculty in effective teaching practices; and recognizing time spent in the redesign of introductory courses or in research on teaching and learning the discipline as evidence of a faculty member’s productivity as a teacher-scholar.