Handbook of Research Synthesis and Meta-Analysis, The

Handbook of Research Synthesis and Meta-Analysis, The

HARRIS COOPER
LARRY V. HEDGES
JEFFREY C. VALENTINE
Copyright Date: 2009
Published by: Russell Sage Foundation
Pages: 632
https://www.jstor.org/stable/10.7758/9781610441384
  • Cite this Item
  • Book Info
    Handbook of Research Synthesis and Meta-Analysis, The
    Book Description:

    When the first edition of The Handbook of Research Synthesis was published in 1994, it quickly became the definitive reference for researchers conducting meta-analyses of existing research in both the social and biological sciences. In this fully revised second edition, editors Harris Cooper, Larry Hedges, and Jeff Valentine present updated versions of the Handbook’s classic chapters, as well as entirely new sections reporting on the most recent, cutting-edge developments in the field. Research synthesis is the practice of systematically distilling and integrating data from a variety of sources in order to draw more reliable conclusions about a given question or topic. The Handbook of Research Synthesis and Meta-Analysis draws upon years of groundbreaking advances that have transformed research synthesis from a narrative craft into an important scientific process in its own right. Cooper, Hedges, and Valentine have assembled leading authorities in the field to guide the reader through every stage of the research synthesis process—problem formulation, literature search and evaluation, statistical integration, and report preparation. The Handbook of Research Synthesis and Meta-Analysis incorporates state-of-the-art techniques from all quantitative synthesis traditions. Distilling a vast technical literature and many informal sources, the Handbook provides a portfolio of the most effective solutions to the problems of quantitative data integration. Among the statistical issues addressed by the authors are the synthesis of non-independent data sets, fixed and random effects methods, the performance of sensitivity analyses and model assessments, and the problem of missing data. The Handbook of Research Synthesis and Meta-Analysis also provides a rich treatment of the non-statistical aspects of research synthesis. Topics include searching the literature, and developing schemes for gathering information from study reports. Those engaged in research synthesis will also find useful advice on how tables, graphs, and narration can be used to provide the most meaningful communication of the results of research synthesis. In addition, the editors address the potentials and limitations of research synthesis, and its future directions. The past decade has been a period of enormous growth in the field of research synthesis. The second edition Handbook thoroughly revises original chapters to assure that the volume remains the most authoritative source of information for researchers undertaking meta-analysis today. In response to the increasing use of research synthesis in the formation of public policy, the second edition includes a new chapter on both the strengths and limitations of research synthesis in policy debates and decisions. Another new chapter looks at computing effect sizes and standard errors from clustered data, such as schools or clinics. Authors also discuss updated techniques for locating hard-to-find “fugitive” literature, ways of systematically assessing the quality of a study, and progress in statistical methods for detecting and estimating the effects of publication bias. The Handbook of Research Synthesis and Meta-Analysis is an illuminating compilation of practical instruction, theory, and problem solving. This unique volume offers the reader comprehensive instruction in the skills necessary to conduct powerful research syntheses meeting the highest standards of objectivity. The significant developments included in the second edition will ensure that the Handbook remains the premier text on research synthesis for years to come.

    eISBN: 978-1-61044-138-4
    Subjects: Sociology, Psychology

Table of Contents

  1. Front Matter
    (pp. i-vi)
  2. Table of Contents
    (pp. vii-x)
  3. FOREWORD
    (pp. xi-xiv)
    Harris Cooper, Larry Hedges and Jeff Valentine

    The past decade has been a time of enormous growth in the need for and use of research syntheses in the social and behavioral sciences. The need was given impetus by the increase in social research that began in the 1960s and continues unabated today. Over the past three decades, methods for the retrieval and analysis of research literatures have undergone enormous change. Literature reviewing used to be a narrative and subjective process. Today, research synthesis has a body of procedural and statistical techniques of its own. Because of the potential pitfalls that confront research synthesists, it is critical that...

  4. ABOUT THE AUTHORS
    (pp. xv-xvi)
  5. PART I INTRODUCTION
    • 1 RESEARCH SYNTHESIS AS A SCIENTIFIC PROCESS
      (pp. 3-16)
      HARRIS COOPER and LARRY V. HEDGES

      The moment we are introduced to science we are told it is a cooperative, cumulative enterprise. Like the artisans who construct a building from blueprints, bricks, and mortar, scientists contribute to a common edifice, called knowledge. Theorists provide our blueprints and researchers collect the data that are our bricks.

      To extend the analogy further yet, we might say that research synthesists are the bricklayers and hodcarriers of the science guild. It is their job to stack the bricks according to plan and apply the mortar that makes the whole thing stick.

      Anyone who has attempted a research synthesis is entitled...

  6. PART II FORMULATING A PROBLEM
    • 2 HYPOTHESES AND PROBLEMS IN RESEARCH SYNTHESIS
      (pp. 19-36)
      HARRIS COOPER

      Texts on research methods often list sources of research ideas (see, for example, Cherulnik 2001). Sometimes ideas for research come from personal experiences, sometimes from pressing social issues. Sometimes a researcher wishes to test a theory meant to help understand the roots of human behavior. Yet other times researchers find topics by reading scholarly research.

      Still, sources of ideas for research are so plentiful that the universe of possibilities seems limitless. Perhaps we must be satisfied with Karl Mannheim’s (1936) suggestion that the ideas researchers pursue are rooted in their social and material relations, from what the researcher deems important...

    • 3 STATISTICAL CONSIDERATIONS
      (pp. 37-48)
      LARRY V. HEDGES

      Research synthesis is an empirical process. As with any empirical research, statistical considerations have an influence at many points in the process. Some of these, such as how to estimate a particular effect parameter or establish its sampling uncertainty, are narrowly matters of statistical practice. They are considered in detail in subsequent chapters of this handbook. Other issues are more conceptual and might best be considered statistical considerations that impinge on general matters of research strategy or interpretation. This chapter addresses selected issues related to interpretation.

      The formulation of the research synthesis problem has important implications for the statistical methods...

  7. PART III SEARCHING THE LITERATURE
    • 4 SCIENTIFIC COMMUNICATION AND LITERATURE RETRIEVAL
      (pp. 51-72)
      HOWARD D. WHITE

      Following analysts such as Russell Ackoff and his colleagues (1976), it is convenient to divide scientific communication into four modes: informal oral, informal written, formal oral, and formal written. The first two, exemplified by telephone conversations and mail among colleagues, are relatively free-form and private. They are undoubtedly important, particularly as ways of sharing news and of receiving preliminary feedback on professional work (Menzel 1968; Garvey and Griffith 1968; Griffith and Miller 1970). Only through the formal modes, however, can scientists achieve their true goal, which is recognition for claims of new knowledge in a cumulative enterprise. The cost to...

    • 5 USING REFERENCE DATABASES
      (pp. 73-102)
      JEFFREY G. REED and PAM M. BAXTER

      The objective of this chapter is to introduce standard literature search tools and their use in research synthesis. Our focus is on research databases relevant to the social, behavioral, and medical sciences. We assume that the reader

      is acquainted with the key literature in his or her own discipline (such as, basic journal publications, handbooks and other reference tools) and the most important sources on the topic of synthesis;

      intends to conduct an exhaustive search of the literature—that is he or she desires to be thorough and identify a comprehensive list of almost all citations and documents that provide...

    • 6 GREY LITERATURE
      (pp. 103-126)
      HANNAH R. ROTHSTEIN and SALLY HOPEWELL

      The objective of this chapter, and of the original in the 1994 edition of this handbook, on which this one is based, is to provide methods of retrieving hard to find literature and information on particular areas of research.¹ We begin by briefly outlining the many developments that have taken place in retrieval of hard-to-find research results over the past decade, including the formalization of this work as a distinct research specialization.

      Although we continue to focus on how to retrieve hard to find literature, developments in research synthesis methodology allow us to place retrieval of this literature in the...

  8. PART IV CODING THE LITERATURE
    • 7 JUDGING THE QUALITY OF PRIMARY RESEARCH
      (pp. 129-146)
      JEFFREY C. VALENTINE

      Empirical studies vary in terms of the rigor with which they are conducted, and the quality of the studies comprising a research synthesis can have an impact on the validity of conclusions arising from that synthesis. However, the wide agreement on these points belies a fundamental tension between two very different approaches in research synthesis to addressing study quality and its impact on the validity of study conclusions. These approaches are the focus of this chapter. In the sections that follow I provide a definition for the termstudy quality, discuss two approaches for addressing study quality in a research...

    • 8 IDENTIFYING INTERESTING VARIABLES AND ANALYSIS OPPORTUNITIES
      (pp. 147-158)
      MARK W. LIPSEY

      Research synthesis relies on information reported in a selection of studies on a topic of interest. The purpose of this chapter is to examine the types of variables that can be coded from those studies and to outline the kinds of relationships that can be examined in the analysis of the resulting data. It identifies a range of analysis opportunities and endeavors to stimulate the synthesist’s thinking about what data might be used in a research synthesis and what sorts of questions might be addressed.

      Research synthesis revolves around the effect sizes that summarize the main findings of each study...

    • 9 SYSTEMATIC CODING
      (pp. 159-176)
      DAVID B. WILSON

      A research synthesist has collected a set of studies that address a similar research question and wishes to code the studies to create a dataset suitable for meta analysis. This task is analogous to interviewing, but a study rather than a person is interviewed. Interrogating might be a more apt description, though it is usually the coder who loses sleep because of the study and not the other way around. The goal of this chapter is to provide synthesists with practical advice on designing and developing a coding protocol suitable to this task. A coding protocol is both the coding...

    • 10 EVALUATING CODING DECISIONS
      (pp. 177-204)
      ROBERT G. ORWIN and JACK L. VEVEA

      Coding is a critical part of research synthesis. It is an attempt to reduce a complex, messy, context-laden, and quantification-resistant reality to a matrix of numbers. Thus it will always remain a challenge to fit the numerical scheme to the reality, and the fit will never be perfect. Systematic strategies for evaluating coding decisions enable the synthesist to control for much of the error inherent in the process. When used in conjunction with other strategies, they can help reduce error as well. This chapter discusses strategies to reduce error as well as those to control for error and suggests further...

  9. PART V STATISTICALLY DESCRIBING STUDY OUTCOMES
    • 11 VOTE-COUNTING PROCEDURES IN META-ANALYSIS
      (pp. 207-220)
      BRAD J. BUSHMAN and MORGAN C. WANG

      As the number of scientific studies continues to grow, it becomes increasingly important to integrate the results from these studies. One simple approach involves counting votes. In the conventional vote-counting procedure, one simply divides studies into three categories: those with significant positive results, those with significant negative results, and those with nonsignificant results. The category containing the most studies is declared the winner. For example, if the majority of studies examining a treatment found significant positive results, then the treatment is considered to have a positive effect.

      Many authors consider the conventional vote-counting procedure to be crude, flawed, and worthless...

    • 12 EFFECT SIZES FOR CONTINUOUS DATA
      (pp. 221-236)
      MICHAEL BORENSTEIN

      In any meta-analysis, we start with summary data from each study and use it to compute an effect size for the study. An effect size is a number that reflects the magnitude of the relationship between two variables. For example, if a study reports the mean and standard deviation for the treated and control groups, we might compute the standardized mean difference between groups. Or, if a study reports events and nonevents in two groups we might compute an odds ratio. It is these effect sizes that are then compared and combined in the meta-analysis.

      Consider figure 12.1, the forest...

    • 13 EFFECT SIZES FOR DICHOTOMOUS DATA
      (pp. 237-254)
      JOSEPH L. FLEISS and JESSE A. BERLIN

      In many studies measurements are made on binary (dichotomous) rather than numerical scales. Examples include studies of attitudes or opinions (the two categories for the response variable being agree or disagree with some statement), case-control studies in epidemiology (the two categories being exposed or not exposed to some hypothesized risk factor), and intervention studies (the two categories being improved or unimproved or, in studies of medical interventions, experiencing a negative event or not). In this chapter we present and analyze four popular measures of association or effect appropriate for categorical data: the difference between two probabilities, the ratio of two...

  10. PART VI STATISTICALLY COMBINING EFFECT SIZES
    • 14 COMBINING ESTIMATES OF EFFECT SIZE
      (pp. 257-278)
      WILLIAM R. SHADISH and C. KEITH HADDOCK

      In 1896, Sir Almroth Wright—a colleague and mentor of Sir Alexander Fleming, who discovered penicillin—developed a vaccine to protect against typhoid (Susser 1977; see Roberts 1989). The typhoid vaccine was tested in several settings, and on the basis of these tests the vaccine was recommended for routine use in the British army for soldiers at risk for the disease. In that same year, Karl Pearson, the famous biometrician, was asked to examine the empirical evidence bearing on the decision. To do so, he synthesized evidence from five studies reporting data about the relationship between inoculation status and typhoid...

    • 15 ANALYZING EFFECT SIZES: FIXED-EFFECTS MODELS
      (pp. 279-294)
      SPYROS KONSTANTOPOULOS and LARRY V. HEDGES

      A central question in research synthesis is whether methodological, contextual, or substantive differences in research studies are related to variation in effect-size parameters.

      Both fixed- and random-effects statistical methods are available for studying the variation in effects. The choice of which to use is sometimes a contentious issue in both meta-analysis as well as primary analysis of data. The choice of statistical procedures should primarily be determined by the kinds of inference the synthesist wishes to make. Two different inference models are available, sometimes calledconditionalandunconditionalinference (see Hedges and Vevea 1998). The conditional model attempts to make...

    • 16 ANALYZING EFFECT SIZES: RANDOM-EFFECTS MODELS
      (pp. 295-316)
      STEPHEN W. RAUDENBUSH

      This volume considers the problem of quantitatively summarizing results from a stream of studies, each testing a common hypothesis. In the simplest case, each study yields a single estimate of the impact of some intervention. Such an estimate will deviate from the true effect size as a function of random error because each study uses a finite sample size. What is distinctive about this chapter is that the true effect size itself is regarded as a random variable taking on different values in different studies, based on the belief that differences between the studies generate differences in the true effect...

    • 17 CORRECTING FOR THE DISTORTING EFFECTS OF STUDY ARTIFACTS IN META-ANALYSIS
      (pp. 317-334)
      FRANK L. SCHMIDT, HUY LE and IN-SUE OH

      Every study has imperfections, many of which bias the results. In some cases we can define precisely what a methodologically ideal study would be like, and thus say that the effect size obtained from any real study will differ to some extent from the value that would have been obtained had the study been methodologically perfect. Although it is important to estimate and eliminate bias in individual studies, it is even more important to remove such errors in research syntheses such as meta-analyses.

      Some authors have argued that meta-analysts should not correct for study imperfections because the purpose of meta-analysis...

  11. PART VII SPECIAL STATISTICAL ISSUES AND PROBLEMS
    • 18 EFFECT SIZES IN NESTED DESIGNS
      (pp. 337-356)
      LARRY V. HEDGES

      Studies with nested designs are frequently used to evaluate the effects of social treatments, such as interventions, products, or technologies in education or public health. One common nested design assigns entire sites—often classrooms, schools, clinics, or communities—to the same treatment group, with different sites assigned to different treatments. Experiments with designs of this type are also calledgroup randomizedorcluster randomizeddesigns because sites such as schools or communities correspond to statistical clusters. In experimental design terminology, these designs are designs involving clusters as nested factors. Nested factors are groupings of individuals that occur only within one...

    • 19 STOCHASTICALLY DEPENDENT EFFECT SIZES
      (pp. 357-376)
      LEON J. GLESER and INGRAM OLKIN

      Much of the literature on meta-analysis deals with analyzing effect sizes obtained from k independent studies in each of which a single treatment is compared with a control (or with a standard treatment). Because the studies are statistically independent, so are the effect sizes.

      Studies, however, are not always so simple. For example, some may compare multiple variants of a type of treatment against a common control. Thus, in a study of the beneficial effects of exercise on blood pressure, independent groups of subjects may each be assigned one of several types of exercise: running for twenty minutes daily, running...

    • 20 MODEL-BASED META-ANALYSIS
      (pp. 377-396)
      BETSY JANE BECKER

      In this chapter I introduce model-based meta-analysis and update what is known about research syntheses aimed at examining models and questions more complex than those addressed in typical bivariate meta-analyses. I begin with the idea of model-based (or model-driven) meta-analysis and the related concept of linked meta-analysis, and describe the benefits (and limitations) of model-based meta-analysis. In discussing how to do a model-based meta-analysis, I pay particular attention to the practical concerns that differ from those relevant to a typical bivariate meta-analysis. To close, I provide a brief summary of research that has been done on methods for model-driven meta-analysis...

  12. PART VIII DATA INTERPRETATION
    • 21 HANDLING MISSING DATA
      (pp. 399-416)
      THERESE D. PIGOTT

      This chapter discusses what researchers can do when studies are missing the information needed for meta-analysis. Despite careful evaluation of coding decisions, researchers will find that studies in a research synthesis invariably differ in the types and quality of the information reported. Here I examine the types of missing data that occur in a research synthesis, and discuss strategies synthesists can use when faced with missing data.

      Problems caused by missing data can never be entirely alleviated. As such, the first strategy for addressing missing data should be to contact the study authors. Doing so remains a viable strategy in...

    • 22 SENSITIVITY ANALYSIS AND DIAGNOSTICS
      (pp. 417-434)
      JOEL B. GREENHOUSE and SATISH IYENGAR

      At every step in a research synthesis, decisions are made that can affect the conclusions and inferences drawn from that analysis. Sometimes a decision is easy to defend, such as one to omit a poor quality study from the meta-analysis based on prespecified exclusion criteria, or to use a weighted average instead of an unweighted one to estimate an effect size. At other times, the decision is less convincing, such as that to use only the published literature, to omit a study with an unusual effect size estimate, or to use a fixed effects instead of a random effects model....

    • 23 PUBLICATION BIAS
      (pp. 435-452)
      ALEX J. SUTTON

      That the published scientific literature documents only a proportion of the results of all research carried out is a perpetual concern. Further, there is good direct and indirect evidence to suggest that the unpublished proportion may be systematically different from the published, since selectivity may exist in deciding what to publish. (Song et al. 2000; Dickersin 2005) For example, researchers may choose not to write up and submit studies with uninteresting findings (such as nonstatistically significant effect sizes), or such studies may not be accepted for publication. Even if a study is published, there may be selectivity in which aspects...

  13. PART IX TYING RESEARCH SYNTHESIS TO SUBSTANTIVE ISSUES
    • 24 ADVANTAGES OF CERTAINTY AND UNCERTAINTY
      (pp. 455-472)
      WENDY WOOD and ALICE H. EAGLY

      A research synthesis typically is not an endpoint in the investigation of a topic. Rarely does a synthesis offer a definitive answer to the theoretical or empirical question that inspired the investigation. Instead, most research syntheses serve as way stations along a sometimes winding research path. The goal is to describe the status of a research literature by highlighting what is unknown as well as what is known. It is the unknowns that are especially likely to suggest useful directions for new research.

      The purpose of this chapter is to delineate the possible relations between research syntheses, theory development, and...

    • 25 RESEARCH SYNTHESIS AND PUBLIC POLICY
      (pp. 473-494)
      DAVID S. CORDRAY and PAUL MORPHY

      These three statements are based on some form of research syntheses. Each was motivated by different interests but all share a purpose—they were conducted to inform policy makers and practitioners about what works. Rather than relying on personal opinion, a consensus of experts, or available research, each statement is based on the results of a systematic, quantitatively based synthesis. The purpose of this chapter is to describe the past, present and future roles of research synthesis in public policy arenas like the ones represented in our three examples. Because meta-analysis is the chief research synthesis method for quantitatively integrating...

  14. PART X REPORTING THE RESULTS
    • 26 VISUAL AND NARRATIVE INTERPRETATION
      (pp. 497-520)
      GEOFFREY D. BORMAN and JEFFREY A. GRIGG

      Meta-analyses in the social sciences and education often provide policy makers, practitioners, and researchers with critical new information that summarizes the central quantitative findings from a particular research literature. However, meta-analytic articles and reports are also rather technical pieces of work that can leave many readers without a clear sense of the results and their implications. In this respect, both visual and narrative interpretations of research syntheses are important considerations for presenting and describing complex results in more informative and accessible ways.

      Typically, readers of meta-analyses in the social sciences and education are offered two types of summary tables to...

    • 27 REPORTING FORMAT
      (pp. 521-534)
      MIKE CLARKE

      The other chapters in this handbook discuss the complexities and intricacies of conceptualizing and conducting research syntheses or systematic reviews. If these pieces of research are to be of most use to readers, they need to be reported in a clear way. The authors should provide a level of detail that allows readers to assess the methods used for the review, as well as the robustness and reliability of its findings, any statistical analyses, and the conclusions. If readers have come to the review for information to help make a decision about future action, they need to be presented with...

  15. PART XI SUMMARY
    • 28 THREATS TO THE VALIDITY OF GENERALIZED INFERENCES
      (pp. 537-560)
      GEORG E. MATT and THOMAS D. COOK

      This chapter provides a nonstatistical way of summarizing many of the main points in the preceding chapters. In particular, it takes the major assumptions outlined and translates them from formal statistical notation into ordinary English. The emphasis is on expressing specific violations of formal meta-analytic assumptions as concretely labeled threats to valid inference. This explicitly integrates statistical approaches to meta-analysis with a falsificationist framework that stresses how secure knowledge depends on ruling out alternative interpretations. Thus, we aim to refocus readers’ attention on the major rationales for research synthesis and the kinds of knowledge meta-analysts seek to achieve.

      The special...

    • 29 POTENTIALS AND LIMITATIONS
      (pp. 561-572)
      HARRIS COOPER and LARRY V. HEDGES

      We suspect that readers of this volume have reacted to it in one of two ways: they have been overwhelmed by the number and complexity of the issues that face a research synthesist or they have been delighted to have a manual to help them through the synthesis process. We further suspect that which reaction it was depended on whether the book was read while thinking about or while actually performing a synthesis. In the abstract, the concerns raised in this handbook must seem daunting. Concretely, however, research syntheses have been carried out for decades and will be for decades...

  16. GLOSSARY
    (pp. 573-584)
  17. APPENDIX A DATA SETS
    (pp. 585-590)
  18. APPENDIX B TABLES
    (pp. 591-594)
  19. INDEX
    (pp. 595-616)