Future of Meta-Analysis, The

Future of Meta-Analysis, The

Kenneth W. Wachter
Miron L. Straf
Copyright Date: 1990
Published by: Russell Sage Foundation
Pages: 248
  • Cite this Item
  • Book Info
    Future of Meta-Analysis, The
    Book Description:

    Scientific progress often begins with the difficult task of preparing informed, conclusive reviews of existing research. Since the 1970s, the traditional "subjective" approach to research reviewing in the social sciences has been challenged by a statistical alternative known as meta-analysis. Meta-analysis provides a principled method of distilling reliable generalizations from previous studies on a single topic, thereby providing a quantitative and objective background for future research.

    The Future of Meta-Analysisbrings together expert researchers for an in-depth examination of this new methodology-not to promote a consensus view but rather to explore from several perspectives the theories, tensions, and concerns of meta-analysis, and to illustrate through concrete examples the rationale behind meta-analytic decisions.

    In a meta-analysis prepared especially for this volume, a statistician and a psychologist review the existing literature on aphasia treatment. In a second study, experts analyze six still-unpublished meta-analyses sponsored by the National Institute of Education to investigate the effects of school desegregation on the academic achievement of black children. This unique case study approach provides valuable discussion of the process of meta-analysis and of the current implications of meta-analysis for policy assessment.

    Prepared under the auspices of the National Research Council,The Future of Meta-Analysispresents a forum for leaders in this rapidly evolving field to discuss salient conceptual and technical issues and to offer a new theoretical framework, further methodological guidance, and statistical innovations that anticipate a future in which meta-analysis will play an even more effective and valuable role in social science research.

    eISBN: 978-1-61044-546-7
    Subjects: Psychology, Sociology

Table of Contents

  1. Front Matter
    (pp. i-iv)
  2. Workshop on the Future of Meta-Analysis
    (pp. v-vi)
  3. Committee on National Statistics
    (pp. vii-viii)
  4. Acknowledgments
    (pp. ix-x)
  5. Table of Contents
    (pp. xi-xii)
  6. Introduction
    (pp. xiii-xxviii)
    Kenneth W. Wachter and Miron L. Straf

    In October 1986 the Committee on National Statistics (CNSTAT) of the National Research Council convened a workshop on the future of meta-analysis. The goal of the workshop was to assess the role actually played by meta-analytic methodologies in current practice and, in particular, to assess their applicability to policy-relevant research, identifying strengths and limitations and suggesting priorities for future research. The presentations at that workshop, the transcript of key parts of the discussion, and the subsequent comments by workshop members on new ideas from the workshop form the basis for the present volume.

    Meta-analysis refers to the application of quantitative...

  7. I Prospects
    • 1 History and Goals
      (pp. 3-10)
      Ingram Olkin

      We have reached a point where psychologists, sociologists, educators, and medical researchers are all of the opinion that meta-analysis was first developed in their particular discipline. It is well known that as soon as you say that you are first, you are bound to be wrong. So I start with some history.

      I suspect that the integration of results of independent evidence is quite old. Certainly, there is synthesis in every jury trial, in the deliberations of a committee in awarding a prize, and in the formulation of policy. Scientists have also required synthesis, and it may be of interest...

    • 2 Directions for Future Methodology
      (pp. 11-26)
      Larry V. Hedges

      I believe that the most significant contribution of meta-analysis is to have focused attention on methodologicalrigorin research reviewing. Methodological standards in original research help ensure the validity of the research. They exist because it is known that some methodological procedures are subject to biases that render research results invalid or at least uninterpretable. For example, some methods of problem formulation (e.g., post hoc hypothesis formulation), data collection (e.g., purposefully biased or nonrandom sampling), data evaluation (e.g., eliminating subjects whose behavior contradicts the research hypothesis), data analysis (e.g., failure to use statistical methods to evaluate stochastic evidence), and reporting...

  8. II Case Study:: The Effects of Rehabilitation Therapy for Aphasia
    • 3 The Making of a Meta-Analysis: A Quantitative Review of the Aphasia Treatment Literature
      (pp. 29-46)
      Joel B. Greenhouse, Davida Fromm, Satish Iyengar, Mary Amanda Dew, Audrey L. Holland and Robert E. Kass

      The project described here began in 1985, when Audrey Holland and Davida Fromm, of the Western Psychiatric Institute and Clinic, University of Pittsburgh, who had become interested in meta-analysis as a primary research tool, invited Joel Greenhouse, of the Statistics Department at Carnegie Mellon University, to collaborate with them on a quantitative review of the aphasia treatment literature. A working group consisting of three statisticians (Joel Greenhouse, Satish Iyengar, and Robert Kass), two speech pathologists (Davida Fromm and Audrey Holland), and a social psychologist (Mary Amanda Dew) was already formed and meeting when Stephen Fienberg and Kenneth Wachter, of the...

    • 4 A Discussion of the Aphasia Study
      (pp. 47-52)
      Nan M. Laird

      The original Committee on National Statistics (CNSTAT) idea in planning the workshop on the future of meta-analysis was to involve a number of methodologists and experienced meta-analysts by asking them to prepare meta-analysis case studies. My position on this plan was that we were naive to expect these people to donate their time, not to mention that of their students and colleagues. In the beginning, my opinion did not receive much support; however, subsequent CNSTAT meetings generally bore out my predictions. In the end, only one group of researchers was willing to undertake the project, and probably only because they...

  9. III Case Study:: The Effects of School Desegregation on the Academic Achievement of Black Children
    • 5 Research, Meta-Analysis, and Desegregation Policy
      (pp. 55-60)
      Jeffrey M. Schneider

      Although much of the research literature on school desegregation has focused specifically on determining the effects of desegregation on academic performance of black students, no one has conclusively answered the question. Reputable social scientists do find statistically significant relationships (positive and negative) which persist even after controlling for socioeconomic status (SES), sex, and other relevant factors, but inconclusive research findings have led to differing conceptions of the size and impact of the effect. Perhaps desegregation has important effects on black student achievement, but the nature of these effects may vary with type of student, type of school, type of community,...

    • 6 An Overview of the Desegregation Meta-Analyses
      (pp. 61-70)
      Linda Ingram

      This chapter summarizes the findings of the scholars selected by the National Institute of Education (NIE) to review desegregation research (U.S. Department of Education 1984) in order to make the methodological discussions of this case study in Chapters 7 to 10 more comprehensible. A full account of the studies is beyond our scope, but a few brief remarks should be useful. Jeffrey Schneider has described the process by which these scholars were selected and how they went about selecting a common set of studies to review. Here, the work of each team of researchers is summarized, along with Thomas Cook’s...

    • 7 Comments on the Desegregation Summary Analysis
      (pp. 71-74)
      S. James Press

      Thomas Cook presented an interesting summary of several distinct meta-analyses of school desegregation studies carried out by Armor, Crain, Miller, Stephan, Walbert, and Wortman. He found positive and neutral effects of school desegregation on black student achievement. But let us examine the bases of his conclusions.

      The author focused on measures of central tendency of effect sizes in the various meta-analyses. He examined means, medians, and modes of effect sizes in the analyses; and that may be all that can be done given the information available to him. Nevertheless, it is hard to find any discussion of quantitative inference being...

    • 8 On the Social Psychology of Using Research Reviews
      (pp. 75-88)
      Harris M. Cooper

      The objective of this research was to examine naturalistically how literature reviews are carried out and how they are evaluated by interested readers. Obviously, the convening of NIE’s panel on desegregation and black achievement provided a rare opportunity for studying the process of research synthesis. Six expert researchers were asked to draw conclusions about a single hypothesis using a nearly common set of studies. Both the Desegregation Studies Team and the panelists agreed to take part in my research. It is the outcome of this research that will be reported in this paper.

      Because of the structure of the panel’s...

    • 9 Moving Beyond Meta-Analysis
      (pp. 89-98)
      Harris M. Cooper

      In this chapter I would like to expand the remarks I made in “On the Social Psychology of Using Research Reviews: The Case of Desegregation and Black Achievement” in three directions. To begin, I would like to examine the question of how quantitative estimates of effect size are given substantive meaning.

      In Thomas Cook’s summary (1984) of the meta-analyst’s findings, he concluded that the estimates of the magnitude of the desegregation effect varied within a remarkably small range. His calculations revealed that all panelists agreed that desegregation did not cause a decrease in black achievement and that the mean gain...

    • 10 An Assessment from the Policy Perspective
      (pp. 99-120)
      David S. Cordray

      Although the statistical foundations for meta-analysis can be traced back at least 50 years, the idea of quantitatively combining the results of independent studies is relatively new within the social and behavioral sciences. Like many new ideas, it has been the focus of much debate. Shortly after Glass (1976) outlined the general tactics and conditions for summarizing evidence across studies, a series of critiques appeared in the literature. For some critics, meta-analysis was viewed as an unsatisfactory approach to understanding the cumulative effects of interventions; Eysenck (1978) went as far as characterizing it as an exercise in “mega-silliness.” Others have...

  10. IV Vantage Points
    • 11 An Evaluation of Procedures and Results
      (pp. 123-134)
      Robert Rosenthal

      In the years 1980, 1981, and 1982 alone, well over 300 papers were published on the topic of meta-analysis (Lamb and Whitla 1983). Does this represent a giant stride forward in the development of the behavioral and social sciences or does it signal a lemming-like flight to disaster? Judging from reactions to past meta-analytic enterprises, there are at least some who take the more pessimistic view. Some three dozen scholars were invited to respond to a meta-analysis of studies of interpersonal expectancy effects conducted by Don Rubin and me (Rosenthal and Rubin 1978). Although much of the commentary dealt with...

    • 12 A Survey Perspective
      (pp. 135-138)
      Norman M. Bradburn

      At this workshop, doing a meta-analysis has been compared to conducting empirical research in which one wants to treat the data quantitatively. I believe that it is similar to designing a survey except that the units of analysis are studies rather than individuals. The same decisions required in designing a good survey are required in designing a meta-analysis.

      My definition of a meta-analysis is a study that summarizes the literature of a particular subject area in a quantitative way. In order to do this, one must devise a metric that can be applied across all units and achieve comparability among...

    • 13 Methodological Observations on Bias
      (pp. 139-152)
      Fredric M. Wolf

      There have been increasing advances in the application of quantitative methods to literature reviewing. All innovations in method or application hold both promise and problems, and meta-analysis is no different in this regard. The advancement of knowledge is a historical, cumulative process. Trial-and-error and serendipity, as well as planfulness, have played roles in the evolution of the scientific method in advancing knowledge. Indeed, the “self-correcting” nature of science was recognized from the onset as one of its principal advantages over what Charles Peirce (1968 [1877]) considered the other three ways of knowing (tenacity, authority, and intuition).

      Tenacity, authority, intuition, and...

  11. V Where Do We Go From Here?
    • 14 A New Perspective
      (pp. 155-166)
      Donald B. Rubin

      These comments are designed to be provocative and thereby stimulate new directions for research in meta-analysis. Often, the right way to add provocative stimuli is to claim that everything everybody is doing is wrong. Even when this isn’t true (and it certainly can’t be in this context), it is often useful to take such an attitude and see how far it can be pushed. So I begin by claiming that everything everybody’s been doing statistically for meta-analysis, including the things that Bob Rosenthal and I have done and do, are irrelevant and have missed the point. We all should really...

    • 15 Concepts Under Scrutiny: Discussion
      (pp. 167-184)

      This chapter presents transcripts of four of the discussions that took place at the workshop in West Virginia. The topics covered are (1) completeness of search, (2) main effects, (3) the desegregation studies, and (4) Rubin’s response surface model. The transcripts have been edited, but were left close to their original form in order to convey some of the excitement at the meeting.

      MOSTELLER: The point I’d like to make deals with the question we’ve all talked about here. Should we be trying for completeness? Why are we trying to get all these data? Why shouldn’t we have some statistical...

    • 16 Summing Up
      (pp. 185-190)
      Frederick Mosteller

      I enjoyed the workshop that led to this volume, because the ideas have been well thought out. The two case studies, although they do not appear in my summing up, played an outstanding role in the development of the ideas.

      In summing up, I have not always tried to mention who was responsible for an idea—often authors are multiply responsible. I organized my summary by tasks that the Committee on National Statistics (CNSTAT) and others promoting the future of meta-analysis might want to take up.

      To sum up the summary: first, we need to provide guidelines for carrying out...

  12. References
    (pp. 191-202)
  13. Index
    (pp. 203-210)