Karen S. Hollweg

One of the main goals of this project was to produce a bibliography of the available literature and completed research regarding the influence of the National Science Education Standards (NSES). The purpose of this chapter is to document the methodology used in creating the bibliography (published in Chapter 8) and to give an overview of the information contained in it.

There has been an increasing interest internationally in using research evidence to inform the development of policy and practice. Researchers in the United Kingdom at the Cochrane Collaboration, the Centre for Reviews and Dissemination in York, and the Institute of Education at the University of London have become known for their high-quality systematic reviews of research relevant to education. In 2001, the Evidence Informed Policy and Practice Information and Co-ordination Centre (EPPI-Centre) at the University of London wrote a Review Group Manual to guide the work of individuals interested in participating in their production and dissemination of systematic reviews in education. The manual, available online at http://eppi.ioe.ac.uk, and the Framework, presented as Figure 1-1 in Chapter 1 and first published in Investigating the Influence of Standards (National Research Council [NRC], 2002), served as starting points for this project. The Steering Committee and staff drew on both documents as they designed the literature search and the guidelines for the work of the commissioned authors, described below.

The project Steering Committee and staff wanted to locate as much of the research that addressed the charge as possible and at the same time avoid bias in the search. To make the search as rigorous, exhaustive, and replicable as possible given the limited resources available, two basic strategies were employed: (1) electronic searches of bibliographic databases, journals, and federally funded agencies and institutions, and (2) searches of Web sites of numerous organizations and agencies actively involved in science education research and analysis of standards-based policies and practices.

First, the NRC library staff performed the research using the following databases: ERIC, NTIS, PAIS, PsycINFO, and Sociological Abstracts. In simultaneous searches of these databases, the librarian created a large base set consisting of documents produced between 1993 and 2001 by the journals, federal agencies, and

organizations listed in Box 7-1. This base set was then cross-searched using the keywords in Box 7-2. Both of these lists were generated by a combination of suggestions from members of the Steering Committee, the Committee on Science Education K-12, staff, and others consulted by staff. The goal was to search multiple sources representing the full range of large and small entities involved in standards-based science education work and to include the work of groups having different philosophical and political perspectives.

Keyword searches were supplemented by “free text” searches—that is, looking through titles and abstracts for key words and phrases. To prevent exclusion of potentially useful studies, the searches were intentionally overinclusive (e.g., including full text, rather than just titles and abstracts) and encompassed everything from January 1993 (the year in which the Benchmarks were published) through October 2001.

Knowing that research regarding the NSES and Benchmarks was ongoing, the project staff also attempted to collect “gray” or “fugitive” literature that had not yet been published in journals or other hard copy formats. The primary strategy for this search was accessing and scanning items posted on the Web sites of the agencies and organizations listed in Box 7-1. In addition, science education researchers and officials responsible for managing government-funded research and evaluation programs were contacted and asked to suggest additional sources of material for consideration.

When duplicates were deleted, these searches resulted in several hundred items concerning the NSES and the Benchmarks for Science Literacy.¹

FIGURE 7-1 Bibliography worksheet.

The next step was to identify the items from this large collection that would provide evidence to address the research question: What does the research tell us about the influence of the NSES on various facets of the educational system, on opportunities for all students to learn, and on student learning? Explicit criteria for inclusion were defined and applied to each study to verify that the study actually addressed the research question. Only studies that met inclusion criteria were to be included in the bibliography and provided to the commissioned authors.

To reduce bias in this process, a Bibliography Worksheet was created that defined explicit criteria for inclusion (Figure 7-1). Full documents were obtained for all items that included reference to the NSES or Benchmarks and one or more other key words used in the search, and a copy of the inclusion criteria chart was

attached to each. Either the project director, program officer, or a project intern (graduate student) scanned each document, noted the study design (i.e., criteria 2) and component of the system addressed (criteria 3) by highlighting the pertinent section(s) of the Worksheet, and categorized each document/study as I (meeting the criteria—documents that met at least one of the criteria in 2 and in 3), II (questionable, unclear), or III (not meeting criteria—for documents that did not meet at least one of the criteria in 2 and in 3). The project director reviewed all IIs and assigned them to the I or III categories, erring on the side of overinclusion to prevent exclusion of potentially useful studies. Many of the items categorized as III were fact sheets and classroom activities keyed to the NSES as opposed to studies that assess the NSES as a means of inducing change or that focused on outcomes of standards-based interventions (see criteria 2). The resulting 245 items included implementation or outcome studies that focused on one or more of the elements in the Framework shown on the Bibliography Worksheet.

Each of the commissioned authors was sent copies of the papers that the staff had categorized as relevant to that author’s topic. In addition, staff assigned to each author the responsibility for annotating a number of papers. In general, the paper was assigned to the author for whom the study was most relevant, but studies addressing multiple components of the education system were distributed to equalize the load among authors.

The commissioned authors agreed to evaluate the bibliographic entries relevant to their topics and to write each annotation to include the following:

1. A statement regarding the nature of the work, whether the paper describes conceptual or experimental research, and the type(s) of data used by the researcher(s)

2. The overall purpose of the paper, including methods the researchers used to collect and evaluate that data;

3. The methodological rigor of the research enterprise;

4. The inferences that were drawn;

5. A statement regarding the findings in terms of the areas of influence listed in the inclusion criteria.

Authors were encouraged to add other studies with which they were familiar to the original set of 245 items identified so that the project could provide a more comprehensive bibliography to the field.

The next chapter contains the entire bibliography for the project, including (1) all 245 items identified through the literature search and processed using the inclusion criteria, (2) additional studies that were either published after the search or added by the authors, and (3) references that are cited in this publication for background, but that do not provide research evidence regarding the influence of the NSES.

Annotations are included for the research studies that authors discuss in their review papers and that ground their arguments and conclusions. In cases where a series of studies are included, the most recent one is annotated and earlier ones are mentioned in that annotation. While all annotations have been written using the same guidelines (as noted above), they vary in style and length due to the fact that many different people wrote them. The authors’ rationale explaining how studies were singled out for inclusion in their reviews is contained within each author’s paper and is not part of the bibliography.