Handbook of Research Synthesis, The

Handbook of Research Synthesis, The

HARRIS COOPER
LARRY V. HEDGES
Copyright Date: 1994
Published by: Russell Sage Foundation
Pages: 592
https://www.jstor.org/stable/10.7758/9781610441377
  • Cite this Item
  • Book Info
    Handbook of Research Synthesis, The
    Book Description:

    The Handbook of Research Synthesis is the definitive reference and how-to manual for behavioral and medical scientists applying the craft of research synthesis. It draws upon twenty years of ground-breaking advances that have transformed the practice of synthesizing research literature from an art into a scientific process in its own right. Editors Harris Cooper and Larry V. Hedges have brought together leading authorities to guide the reader through every stage of the research synthesis process—problem formulation, literature search and evaluation, statistical integration, and report preparation. The Handbook of Research Synthesis incorporates in a single volume state-of-the-art techniques from all quantitative synthesis traditions, including Bayesian inference and the meta-analytic approaches. Distilling a vast technical literature and many informal sources, the Handbook provides a portfolio of the most effective solutions to problems of quantitative data integration. The Handbook of Research Synthesis also provides a rich treatment of the non-statistical aspects of research synthesis. Topics include searching the literature, managing reference databases and registries, and developing coding schemes. Those engaged in research synthesis will also find useful advice on how tables, graphs, and narration can be deployed to provide the most meaningful communication of the results of research synthesis. The Handbook of Research Synthesis is an illuminating compilation of practical instruction, theory, and problem solving. It provides an accumulation of knowledge about the craft of reviewing a scientific literature that can be found in no other single source. The Handbook offers the reader thorough instruction in the skills necessary to conduct powerful research syntheses meeting the highest standards of objectivity, systematicity, and rigor demanded of scientific enquiry. This definitive work will represent the state of the art in research synthesis for years to come.

    eISBN: 978-1-61044-137-7
    Subjects: Sociology, Psychology

Table of Contents

  1. Front Matter
    (pp. i-vi)
  2. Table of Contents
    (pp. vii-x)
  3. FOREWORD
    (pp. xi-xiv)
    Harris Cooper and Larry V. Hedges

    TheHandbook of Research Synthesisis an attempt to bring together in a single volume state-of-the-art descriptions of every phase of the research synthesis process, from problem formulation to report writing.

    Determining the content of theHandbookrequired us to use multiple decision criteria. First, by relying on personal experience and knowledge of previous books in the area, we identified the central decision points in conducting a research synthesis. This led to the creation of several of the major divisions of the text as well as specific chapters—for example, vote-counting procedures, combining significance levels, and correcting for sources of...

  4. CONTRIBUTORS
    (pp. xv-xvi)
  5. PART I INTRODUCTION
    • 1 RESEARCH SYNTHESIS AS A SCIENTIFIC ENTERPRISE
      (pp. 3-14)
      HARRIS COOPER and LARRY V. HEDGES

      From the moment we are introduced to science we are told it is a cooperative, cumulative enterprise. Like the artisans who construct a building from blueprints, bricks, and mortar, scientists contribute to a common edifice called knowledge. Theorists provide the blueprints and researchers collect the data that are the bricks.

      To extend the analogy further, we might say that research synthesists are the bricklayers and hodcarriers of the science guild. It is their job to stack the bricks according to plan and apply the mortar that holds the structure together.

      Those who have attempted a research synthesis are entitled to...

  6. PART II FORMULATING A PROBLEM FOR A RESEARCH SYNTHESIS
    • 2 HYPOTHESES AND PROBLEMS IN RESEARCH SYNTHESIS
      (pp. 17-28)
      JUDITH A. HALL, LINDA TICKLE-DEGNEN, ROBERT ROSENTHAL and FREDERICK MOSTELLER

      In the social, behavioral, and medical sciences, investigators wish to strengthen their methods of accumulating evidence. They have been dissatisfied with the classical literature review because these reports of findings are often limited to directions of relationship among variables or to results of significance tests. More and more, reviews of the literature are moving from the traditional literary approaches to the quantitative techniques of research synthesis described in this handbook.

      Whenever more than one study has addressed the same conceptual hypothesis, research synthesis can be done. At the simplest level, an investigator might ask whether a study and its replication...

    • 3 STATISTICAL CONSIDERATIONS
      (pp. 29-38)
      LARRY V. HEDGES

      Research synthesis is an empirical process. As with any empirical research, statistical considerations have an influence at many points in the process. Some of these considerations, such as how to test particular hypotheses, are narrowly matters of statistical practice. They are considered in detail in subsequent chapters of this handbook. Other issues are more conceptual and might best be considered statistical considerations that impinge on general matters of research strategy or interpretation. This chapter addresses selected issues of the latter type.

      The formulation of the research synthesis problem has important implications for the statistical methods that may be appropriate and...

  7. PART III SEARCHING THE LITERATURE
    • 4 SCIENTIFIC COMMUNICATION AND LITERATURE RETRIEVAL
      (pp. 41-56)
      HOWARD D. WHITE

      An essential aspect of science is communication of findings (Ziman 1968). Following analysts such as Ack-off et al. (1976), we can conveniently divide scientific communication into four modes: informal oral, informal written, formal oral, and formal written. The first two, exemplified by telephone conversations and letters among colleagues, are relatively free-form and private. They are undoubtedly important, particularly as ways of sharing news and of receiving preliminary feedback on scholarly work (Menzel 1968; Garvey & Griffith 1968; Griffith & Miller 1970). Only through the formal modes, however, can scientists achieve their true goal, which is recognition for claims of new knowledge in...

    • 5 USING REFERENCE DATABASES
      (pp. 57-70)
      JEFFREY G. REED and PAM M. BAXTER

      The objective of this chapter is to introduce literature search sources and methods that will be of use to the research synthesist. Our focus is on machine-readable databases relevant to the social, behavioral, and medical sciences. We assume that the reader

      1. is familiar with research libraries (e.g., card catalogs, periodical indexes; see Reed & Baxter 1992),

      2. is acquainted with the key literature in his/her discipline and the most important sources on the topic of the synthesis, and

      3. intends to conduct an exhaustive search of the literature.

      The many bibliographic sources that exist take the form of both indexes (providing only bibliographic...

    • 6 RESEARCH REGISTERS
      (pp. 71-84)
      KAY DICKERSIN

      Research synthesis would be a far easier task if there were registers of all initiated research projects. A research register can be defined as a database of research studies, either planned, active, or completed (or any combination of these), usually oriented around a common feature of the studies such as subject matter, funding source, or design. This definition is broad enough to include such resources as a database of research projects approved by an institutional review board, an annual report of a funding body, or a comprehensive bibliographic database such as MEDLINE orPsychological Abstracts. The registers on which this...

    • 7 THE FUGITIVE LITERATURE
      (pp. 85-94)
      MARYLU C. ROSENTHAL

      This chapter describes methods of retrieving hard-to-find (“fugitive”) literature and information on particular areas of research. The elusive information may be about a research report that is unpublished or that has been published but cannot be found through traditional, commonly used sources. Indeed, the information on the results of a study or series of studies may not yet be in any formalwrittenreport but exist instead as data on computer printout along with the researcher’s working notes and memos.

      Literature may be fugitive in a sense other than being “unpublished.” Each year new journals, presenting material in each of...

  8. PART IV EVALUATING THE LITERATURE
    • 8 JUDGING RESEARCH QUALITY
      (pp. 97-110)
      PAUL M. WORTMAN

      The discussion about judging research quality in synthesis dates to Glass’s unveiling of “meta-analysis” in the November 1976 issue of theEducational Researcher(Glass 1976). Prior to that time, the quality of the research studies was considered a major “obstacle” to developing a quantitative synthesis method (Bangert-Drowns 1986). Within a year of Glass’s seminal publication critics were debating the merits of his advice on judging the effect of research quality in meta-analysis (Mansfield & Busse 1977). In his rejoinder to his critics Glass (1978) argued, “The sensible course to follow is to describe—in quantitative terms—features of designs and correlate...

    • 9 IDENTIFYING POTENTIALLY INTERESTING VARIABLES AND ANALYSIS OPPORTUNITIES
      (pp. 111-124)
      MARK W. LIPSEY

      Research synthesis relies on information that is reported in a selection of studies on a topic of interest. The purpose of this chapter is to review the types of variables often available to the synthesist and to outline the kinds of relationships that can be explored in the analysis of the data that results from coding those variables. It will identify a range of opportunities and endeavor to stimulate the synthesist’s thinking about what might go into a research integration and what sorts of questions can be addressed.

      Research synthesis revolves around one or more statistics, called effect sizes, that...

    • 10 SYSTEMATIC CODING FOR RESEARCH SYNTHESIS
      (pp. 125-138)
      WILLIAM A. STOCK

      Research synthesists must make many decisions before the actual coding of studies begins: They must choose the study characteristics of interest, and conventions for coding them, and construct coding forms and a code book. They need to develop methods that ensure the reliable and orderly extraction of information from each report. Because these preparations are iterative, they should allocate adequate time for their completion.

      Coders locate, reduce, manipulate, and transcribe information. Coders may make judgments about study quality and the adequacy of information in study reports. Because coders can make mistakes, the synthesist should subject their codings to independent verification...

    • 11 EVALUATING CODING DECISIONS
      (pp. 139-162)
      ROBERT G. ORWIN

      Coding is a critical part of research synthesis. It represents an attempt to reduce a complex, messy, context-laden, and quantification-resistant reality to a matrix of numbers. Thus, it will always remain a challenge to fit the numerical scheme to the reality, and the fit will never be perfect. Systematic strategies for evaluating coding decisions enable the synthesist to control for much of the error inherent in the process. When used in conjunction with other strategies, they can help reduce error as well. This chapter will discuss strategies to reduce error as well as strategies to control for error and will...

    • 12 METHODS FOR HANDLING MISSING DATA IN RESEARCH SYNTHESIS
      (pp. 163-176)
      THERESE D. PIGOTT

      Previous chapters of this volume have outlined how to define a research question for a research synthesis, develop a comprehensive search for literature, obtain a representative sample of studies, and design a set of coding procedures. Even after a researcher has carefully planned a research synthesis, problems can emerge when studies do not provide comparable information. The problem of missing data occurs in a research synthesis when studies do not report relevant statistics or adequate descriptions of methods needed for applying quantitative techniques for combining results across studies. This chapter will discuss the problems that missing data cause in a...

    • 13 MANAGING META-ANALYTIC DATABASES
      (pp. 177-190)
      GEORGE WOODWORTH

      This chapter addresses topics of interest to specialists who may be involved in collecting and processing data in research syntheses: data coders, data entry personnel, statisticians, data managers, and project directors. In some cases, one person may fill several if not all of these roles. Nevertheless, different and quite distinct skills are required for each function. In this chapter the central player is the data manager; however, since he or she must rely on the data coders to provide data to be managed and must, in turn, deliver data in a useful form to the statistical analyst, there will be...

  9. PART V STATISTICALLY DESCRIBING AND COMBINING STUDIES
    • 14 VOTE-COUNTING PROCEDURES IN META-ANALYSIS
      (pp. 193-214)
      BRAD J. BUSHMAN

      The meta-analyst generally has access to at least one of three types of data from research reports: (a) information that can be used to calculate effect size estimates (e.g., means, standard deviations, test statistic values), (b) information about whether the hypothesis tests found statistically significant relations, and (c) information about the direction of the outcomes. These data are rank ordered, from most to least, in terms of the amount of information they contain.¹ If the first type of data are available, the methods described in Chapters 18-20 of this volume are more appropriate than the methods described in this chapter....

    • 15 COMBINING SIGNIFICANCE LEVELS
      (pp. 215-230)
      BETSY JANE BECKER

      This chapter discusses methods for combining probability values (or significance levels) from independent significance tests. These “combined significance” methods have a long history (e.g., Fisher 1932; Tippett 1931) and have been studied extensively by statisticians. Several of the methods are closely related to the vote-counting techniques described in the preceding chapter.

      The first sections of the chapter describe significance levels and briefly introduce the combined significance methods, differentiating them from parametric methods. Next, the hypotheses tested by the combined significance methods are examined and compared with those tested by the parametric methods. The methods are introduced, with a focus on...

    • 16 PARAMETRIC MEASURES OF EFFECT SIZE
      (pp. 231-244)
      ROBERT ROSENTHAL

      The heart of the enterprise of synthesizing research consists of comparing and combining the results of individual studies of a particular, focused research question. In Chapter 15, the emphasis was on one type of result of an individual study—the test of significance. In the present chapter, the emphasis will be on a different type of result of an individual study—the size of the effect of an independent variable on a dependent variable, or, more generally, the size of the relationship between any two variables (Rosenthal 1991).

      These two types of results, the test of significance (test statistic) and...

    • 17 MEASURES OF EFFECT SIZE FOR CATEGORICAL DATA
      (pp. 245-260)
      JOSEPH L. FLEISS

      In many studies measurements are made on binary rather than numerical scales. Examples include studies of attitudes or opinions (the two categories for the response variable being agree or disagree with some statement), case-control studies in epidemiology (the two categories being exposed or not exposed to some hypothesized risk factor), and intervention studies (the two categories being improved or unimproved). In this chapter we present and analyze four popular measures of association or effect that are appropriate for categorical data: the difference between two probabilities, the ratio of two probabilities, the phi coefficient, and the odds ratio. The odds ratio...

    • 18 COMBINING ESTIMATES OF EFFECT SIZE
      (pp. 261-282)
      WILLIAM R. SHADISH and C. KEITH HADDOCK

      In 1896, Sir Almroth Wright developed a vaccine to protect against typhoid (Susser 1977; Wright, incidentally, was a colleague and mentor of Sir Alexander Fleming, who later discovered penicillin; see Roberts 1989). The typhoid vaccine was tested in several settings, and on the basis of these tests the vaccine was recommended for routine use in the British army for soldiers at risk for the disease. In that same year, Karl Pearson, the famous biometrician, was asked to review the empirical evidence bearing on the decision. To do so, he reviewed evidence from five studies reporting data about the relationship between...

  10. PART VI STATISTICALLY ANALYZING EFFECT SIZES
    • 19 FIXED EFFECTS MODELS
      (pp. 285-300)
      LARRY V. HEDGES

      A central question in research synthesis is whether methodological, contextual, or substantive variations in research studies are related to variations in effect size parameters. Both fixed and random effects methodologies are available for studying the variation in effects. In this chapter, two general classes of fixed effects models are presented. One class of models is appropriate when the independent (study characteristic) variables are categorical. This class of models is analogous to the analysis of variance, but is adapted to the special characteristics of effect size estimates. The second class of models is appropriate for either discrete or continuous independent variables...

    • 20 RANDOM EFFECTS MODELS
      (pp. 301-322)
      STEPHEN W. RAUDENBUSH

      A skilled researcher who wishes to assess the generalizability of findings from a single study will try to select a sample that represents the target population. The researcher will examine interaction effects—for example, by asking whether a new medical treatment depends on the age of the patient; whether the effectiveness of a new method of instruction depends on student aptitude; whether men are as responsive as women to nonverbal communication. If such interactions are null—if treatment effects do not depend on the age, aptitude, or sex of the subject—a finding is viewed as generalizing across these subject...

    • 21 CORRECTING FOR SOURCES OF ARTIFICIAL VARIATION ACROSS STUDIES
      (pp. 323-336)
      JOHN E. HUNTER and FRANK L. SCHMIDT

      Every study has imperfections. In some cases we can define precisely what a perfect study might be, and thus we can say that the effect size value obtained from any real study will differ to some extent from the value that would have been obtained had the study been done perfectly. While it is important to control and estimate bias in isolated studies, it is even more important to reduce such errors in cumulative research reviews such as meta-analysis.

      Some authors have argued that meta-analysts should not correct for study imperfections because the purpose of meta-analysis is only to provide...

  11. PART VII SPECIAL STATISTICAL ISSUES AND PROBLEMS
    • 22 STOCHASTICALLY DEPENDENT EFFECT SIZES
      (pp. 339-356)
      LEON J. GLESER and INGRAM OLKIN

      Much of the literature on meta-analysis deals with analyzing effect sizes obtained fromkindependent studies in each of which a single treatment is compared with a control (or with a standard treatment). Because the studies are statistically independent, so are the effect sizes.

      Studies, however, are not always so simple. Thus, some studies may compare multiple variants of a treatment against a common control. For example, in a study of the beneficial effects of exercise on blood pressure, independent groups of subjects may each be assigned one of several types of exercise: running for 20 minutes daily, running for...

    • 23 EXAMINING EXPLANATORY MODELS THROUGH RESEARCH SYNTHESIS
      (pp. 357-382)
      BETSY JANE BECKER and CHRISTINE M. SCHRAM

      This chapter describes a way to bring substantive theory directly into the process of research synthesis, something that critics of quantitative research synthesis have claimed is missing from it. We begin with a discussion of models and a rationale for incorporating models into research synthesis. A brief history of explanatory modeling in meta-analysis follows. Then we describe three approaches to the analysis of data in a “model-driven” synthesis. Finally, we mention some of the problems that the reviewer faces in conducting a model-driven synthesis and the factors that limit inferences based on models derived from quantitative reviews.

      The term “model”...

    • 24 SENSITIVITY ANALYSIS AND DIAGNOSTICS
      (pp. 383-398)
      JOEL B. GREENHOUSE and SATISH IYENGAR

      At every step in a research synthesis decisions are made that can affect the conclusions and inferences drawn from the analysis. Sometimes, a decision is easy to defend, such as the decision to omit a poor-quality study from the meta-analysis or to use a weighted average instead of an unweighted one to estimate an effect size. At other times, a decision is less defendable, such as the decision to use only the published literature, to omit a study with an unusual effect size estimate, or to use a fixed effects model instead of a random effects model. When the basis...

    • 25 PUBLICATION BIAS
      (pp. 399-410)
      COLIN B. BEGG

      The style of reporting the results of a research study in a journal article is governed as much by human nature as by the tradition of scientific objectivity. That is, research studies are commonly reported in an advocacy style. Statistical significance, if it is achieved, may be used as “proof” of a theory. Moreover, the statistical analysis may be subjectively influenced by the use of a variety of statistical tests, excluding certain categories of subjects, performing analyses in selected subgroups, or adjusting the analysis for covariates, all with the goal of presenting the data in such a way as to...

    • 26 BAYESIAN APPROACHES TO RESEARCH SYNTHESIS
      (pp. 411-422)
      THOMAS A. LOUIS and DANIEL ZELTERMAN

      As in other scientific studies, efficient design, conduct, and analysis of a research synthesis requires that investigators use expert opinion and empirical evidence from previous studies. Design and analysis decisions are made at every stage of the process; these should be based on a combination of subjective and objective inputs. For example, statistical models are based on assumptions that require justification, but the analyst can never prove that the models are correct.

      The Bayesian approach provides a formal structure for incorporating such uncertainties. All unknown parameters are treated as random variables that are governed by a joint probability distribution specified...

  12. PART VIII REPORTING THE RESULTS OF RESEARCH SYNTHESIS
    • 27 THE REPORTING FORMAT
      (pp. 425-438)
      KATHERINE TAYLOR HALVORSEN

      Several articles examining the quality of published research syntheses or meta-analyses have appeared in journals in both the behavioral sciences and medicine (Beaman 1991; Sacks et al. 1987; Jackson 1980). Through these articles and others like them the scientific community has begun to establish guidelines concerning what information about a research synthesis should be made public. It is important to note that a reader’s judgment of the quality of research depends entirely on the clarity of writing and the thoroughness of reporting in the published research report. Becker (1991), in her interviews with review journal editors, found that they “overwhelmingly...

    • 28 THE VISUAL PRESENTATION AND INTERPRETATION OF META-ANALYSES
      (pp. 439-454)
      RICHARD J. LIGHT, JUDITH D. SINGER and JOHN B. WILLETT

      A large proportion of meta-analyses in education and the social sciences, including many that we consider excellent, pose a surprising challenge. The reader must struggle to figure out exactly what the researchers found!

      How are the results of meta-analyses currently presented? Typically, and especially in the social sciences and education, readers are offered two types of summary table. The first table gives a lengthy list of the actual studies that were included in the meta-analysis, together with their date of publication, sample size, effect size, and so forth. The second table presents the overall average effect size for the entire...

  13. PART IX TYING RESEARCH SYNTHESIS TO SUBSTANTIVE ISSUES
    • 29 META-ANALYTIC SYNTHESIS FOR THEORY DEVELOPMENT
      (pp. 457-484)
      NORMAN MILLER and VICKI E. POLLOCK

      In this chapter we contend that some of the features of meta-analysis make it superior to individual studies for testing and formulating theories. In section 2 we discuss the ambiguous use of the term “theory.” In section 3 we discuss what we mean by “testing theory.” In section 4 we review the potential roles of meta-analysis in developing and evaluating theory, which include assessing causal relationships as well as fulfilling other explanatory tasks. In this section we also consider its advantages relative to single empirical studies. In section 5 we discuss three types of meta-analysis that differ in the way...

    • 30 USING RESEARCH SYNTHESES TO PLAN FUTURE RESEARCH
      (pp. 485-500)
      ALICE H. EAGLY and WENDY WOOD

      A quantitative synthesis of research is typically not an endpoint or final step in the investigation of a research topic A formal synthesis rarely offers an integration that is conclusive enough to answer a question decisively. Instead, a meta-analysis more commonly serves as a way station along a sometimes winding route to answering a particular question.

      Of course, a systematic integration of research findings cannot be accomplished at the beginning stage of a field’s investigation and must await a body of studies addressing a hypothesis. However, research syntheses can contribute importantly at both middle and later stages of investigation. By...

  14. PART X SUMMARY
    • 31 THREATS TO THE VALIDITY OF RESEARCH SYNTHESES
      (pp. 503-520)
      GEORG E. MATT and THOMAS D. COOK

      Although the list of validity threats we present is relevant to research synthesis practice, it is not definitive. All threats are empirical products. Any list of threats should change as theories of method are improved and as critical discourse about research synthesis practice accumulates. Some of the threats we discuss apply to conclusions from individual studies as well as from research syntheses. Experienced research synthesists realize that they have to correct inadequacies in individual studies before combining their results across studies (Cooper 1989). Other threats apply only to research syntheses, in that they depend on how estimates from individual studies...

    • 32 POTENTIALS AND LIMITATIONS OF RESEARCH SYNTHESIS
      (pp. 521-530)
      HARRIS COOPER and LARRY V. HEDGES

      We suspect that readers of this volume reacted to it in one of two distinct ways: They may have been overwhelmed by the number and complexity of the issues that face a research synthesist; alternatively, they may have been delighted to have available a manual to help them through the synthesis process. We further suspect that which reaction was experienced depended on whether the book was read while thinking about or while actually performing a synthesis. In the abstract, the concerns raised in this handbook must seem daunting. Concretely however, research syntheses have been carried out for decades and will...

  15. GLOSSARY
    (pp. 531-542)
  16. APPENDIX A: Data Sets
    (pp. 543-548)
  17. APPENDIX B: Tables
    (pp. 549-552)
  18. INDEX
    (pp. 553-576)