Skip to Main Content
Have library access? Log in through your library
Population-Based Survey Experiments

Population-Based Survey Experiments

Diana C. Mutz
Copyright Date: 2011
Edition: STU - Student edition
Pages: 192
  • Cite this Item
  • Book Info
    Population-Based Survey Experiments
    Book Description:

    Population-based survey experiments have become an invaluable tool for social scientists struggling to generalize laboratory-based results, and for survey researchers besieged by uncertainties about causality. Thanks to technological advances in recent years, experiments can now be administered to random samples of the population to which a theory applies. Yet until now, there was no self-contained resource for social scientists seeking a concise and accessible overview of this methodology, its strengths and weaknesses, and the unique challenges it poses for implementation and analysis.

    Drawing on examples from across the social sciences, this book covers everything you need to know to plan, implement, and analyze the results of population-based survey experiments. But it is more than just a "how to" manual. This lively book challenges conventional wisdom about internal and external validity, showing why strong causal claims need not come at the expense of external validity, and how it is now possible to execute experiments remotely using large-scale population samples.

    Designed for social scientists across the disciplines,Population-Based Survey Experimentsprovides the first complete introduction to this methodology.

    Offers the most comprehensive treatment of the subjectFeatures a wealth of examples and practical adviceReexamines issues of internal and external validityCan be used in conjunction with downloadable data from for design and analysis exercises in the classroom

    eISBN: 978-1-4008-4048-9
    Subjects: Sociology

Table of Contents

Export Selected Citations Export to NoodleTools Export to RefWorks Export to EasyBib Export a RIS file (For EndNote, ProCite, Reference Manager, Zotero, Mendeley...) Export a Text file (For BibTex)
  1. Front Matter
    (pp. i-vi)
  2. Table of Contents
    (pp. vii-viii)
  3. List of Tables
    (pp. ix-x)
  4. Preface
    (pp. xi-xvi)
    (pp. 1-22)

    Approaches to scientific knowledge are a bit like rabid sports rivals; often they cannot say anything nice about their own team without simultaneously disparaging the other side. At some level, they know these intense rivalries would not exist if the other team were not a worthy contender, but the positive aspects of the other side are seldom acknowledged.

    Likewise, empirical social scientists tend to develop expertise either in large-scale observational methods such as survey research, or in laboratory-based experimental approaches. They then spend the rest of their careers defending their choice of that particular approach in virtually everything they publish....


    • CHAPTER TWO Treatments to Improve Measurement
      (pp. 25-36)

      The goal of Part I of this volume is to supply numerous examples of different types of population-based experimental treatments from a variety of different social science fields. Chapter 2 begins where population-based experiments commenced, with experimental studies designed to improve measurement of concepts and behaviors that are difficult to quantify accurately. The additional chapters in Part I are organized around five loosely defined categories of treatment, any of which could be put to use to answer research questions in any discipline.

      Chapter 3 describes the most straightforward and transparent of treatment varieties, the direct treatment, as well as progressively...

    • CHAPTER THREE Direct and Indirect Treatments
      (pp. 37-53)

      This chapter covers what are by the far the most widely used and most flexible forms of population-based experimental treatments, direct and indirect treatments. What I call “direct treatments” are those in which the manipulation or intervention is precisely what it appears to be to the participant. Perhaps it is a counter-argument to the respondent’s current position on an issue. Or perhaps the respondent is being asked to judge a photograph of a person. With direct treatments, it is easy for the respondent to see that he/she is being exposed to new information, a new argument, or some other intervention...

    • CHAPTER FOUR Vignette Treatments
      (pp. 54-67)

      One of the advantages of large-scale population-based experiments is the ability to do multi-factor experiments and test complex interactions. The relatively large sample sizes afforded by surveys mean that researchers can, if they so choose, have samples well into the thousands, thereby avoiding embarrassingly small cell sizes in studies with a large number of conditions. This advantage enables researchers to incorporate many experimental treatments simultaneously into a single experiment, and thus affords the opportunity to study complex interactions.

      The vignette approach to treatment described in this chapter is especially well-suited to complex factorial experiments. Although this approach can certainly be...

    • CHAPTER FIVE Treatments in the Context of Games
      (pp. 68-80)

      Yet another way that social scientists have administered experimental treatments is in the context of games, broadly construed. In some cases, these appear to be (or actually are) real games that people play for fun and entertainment. In other cases, these are economic games, which are typically played in a lab setting without an entertainment incentive so much as a fiscal one (thus making them a bit more like work). Such games may or may not have intrinsic entertainment value. In either case, I consider them “games” because the respondent engages in a set of interactions with other players, and...


    • CHAPTER SIX Execution of Population-Based Survey Experiments
      (pp. 83-107)

      Population-based experiments do not fit neatly into pre-existing categories for research designs. For this reason, there is considerable confusion regarding best practices. This chapter addresses a series of practical matters involved in their implementation. The TESS staff learned a great deal from observing the execution of so many varied population-based experiments. Given the creativity and variation in designs, there are few “one size fits all” rules or admonitions for which one cannot find an exception. Nonetheless, there have been some useful lessons learned.

      In this chapter, I hope to eliminate the need for others to learn the hard way. I...

    • CHAPTER SEVEN Analysis of Population-Based Survey Experiments
      (pp. 108-128)

      Investigators tend to approach the analysis and interpretation of population-based experiments from the perspective of usual practices in whatever their home discipline happens to be. Survey researchers analyze their data much as they would a survey, and experimentalists analyze them as they would any experiment. In many respects, this is harmless. However, as it turns out, some of these choices go hand in hand with common errors and faulty assumptions about the most appropriate way to analyze results from population-based experiments. Usual practices are not always appropriate, particularly with disciplines that favor observational methods. This chapter aims to provide guidance...


    • CHAPTER EIGHT External Validity Reconsidered
      (pp. 131-154)

      To date, the main rationale for promoting population-based survey experiments has been external validity. In reflecting on this claim, it became clear to me that things were not so simple as that for several reasons. First, not only was generalizability itself poorly defined, but the received wisdom regarding the generalizability of existing research designs was also highly questionable. In order to sort through the truth and fiction behind these claims and identify the advantages of population-based survey experiments, I begin with a reassessment of the usual heuristics for assessing both internal and external validity.

      As social scientists, most of us...

    • CHAPTER NINE More Than Just Another Method
      (pp. 155-160)

      I admit to taking a certain perverse delight in finding out that some “consensus knowledge” is actually misguided. Sugary breakfast cereals improve kids’ test scores,¹ gentrification attracts minorities,² and despite slang to the contrary, humans are the only primates who lack penis bones.³ Likewise, one of the pleasant side-benefits of population-based survey experiments is that they turn on their head some of the “truisms” learned about research methods while in graduate school. There is no necessary trade-off between internal and external validity in the choice of a research design. Experiments can be useful in evaluating generalizability. Covariates are not always...

  9. Bibliography
    (pp. 161-172)
  10. Index
    (pp. 173-177)