Skip to Main Content
Have library access? Log in through your library
Learning More from Social Experiments

Learning More from Social Experiments: Evolving Analytic Approaches

Howard S. Bloom Editor
Copyright Date: 2005
Published by: Russell Sage Foundation
Pages: 264
  • Cite this Item
  • Book Info
    Learning More from Social Experiments
    Book Description:

    Policy analysis has grown increasingly reliant on the random assignment experiment—a research method whereby participants are sorted by chance into either a program group that is subject to a government policy or program, or a control group that is not. Because the groups are randomly selected, they do not differ from one another systematically. Therefore any differences between the groups at the end of the study can be attributed solely to the influence of the program or policy. But there are many questions that randomized experiments have not been able to address. What component of a social policy made it successful? Did a given program fail because it was designed poorly or because it suffered from low participation rates? In Learning More from Social Experiments, editor Howard Bloom and a team of innovative social researchers profile advancements in the scientific underpinnings of social policy research that can improve randomized experimental studies. Using evaluations of actual social programs as examples, Learning More from Social Experiments makes the case that many of the limitations of random assignment studies can be overcome by combining data from these studies with statistical methods from other research designs. Carolyn Hill, James Riccio, and Bloom profile a new statistical model that allows researchers to pool data from multiple randomized-experiments in order to determine what characteristics of a program made it successful. Lisa Gennetian, Pamela Morris, Johannes Bos, and Bloom discuss how a statistical estimation procedure can be used with experimental data to single out the effects of a program’s intermediate outcomes (e.g., how closely patients in a drug study adhere to the prescribed dosage) on its ultimate outcomes (the health effects of the drug). Sometimes, a social policy has its true effect on communities and not individuals, such as in neighborhood watch programs or public health initiatives. In these cases, researchers must randomly assign treatment to groups or clusters of individuals, but this technique raises different issues than do experiments that randomly assign individuals. Bloom evaluates the properties of cluster randomization, its relevance to different kinds of social programs, and the complications that arise from its use. He pays particular attention to the way in which the movement of individuals into and out of clusters over time complicates the design, execution, and interpretation of a study. Learning More from Social Experiments represents a substantial leap forward in the analysis of social policies. By supplementing theory with applied research examples, this important new book makes the case for enhancing the scope and relevance of social research by combining randomized experiments with non-experimental statistical methods, and it serves as a useful guide for researchers who wish to do so.

    eISBN: 978-1-61044-069-1
    Subjects: Sociology

Table of Contents

Export Selected Citations Export to NoodleTools Export to RefWorks Export to EasyBib Export a RIS file (For EndNote, ProCite, Reference Manager, Zotero, Mendeley...) Export a Text file (For BibTex)
  1. Front Matter
    (pp. i-iv)
  2. Table of Contents
    (pp. v-vi)
  3. Contributors
    (pp. vii-viii)
  4. Preface
    (pp. ix-xii)
    Howard Bloom
  5. Chapter 1 Precedents and Prospects for Randomized Experiments
    (pp. 1-36)
    Charles Michalopoulos

    When families move to low-poverty neighborhoods, their teenage children are less likely to commit crimes (Ludwig, Hirschfield, and Duncan 2001). Couples therapy and family therapy are equally effective at improving marital relationships (Shadish et al. 1995). Increasing welfare benefit amounts by 10 percent discourages 1 percent of low-income parents from working (Burtless 1987). Each of these statements answers a question about the effect, or impact, of a social policy or intervention on people’s behavior. Does helping low-income families move to low-poverty neighborhoods affect their children’s development? Does couples counseling bring more benefits than family therapy? What proportion of low-income parents...

  6. Chapter 2 Modeling Cross-Site Experimental Differences to Find Out Why Program Effectiveness Varies
    (pp. 37-74)
    Howard S. Bloom, Carolyn J. Hill and James A. Riccio

    Charged with planning a new social program, senior administrators in a state human services agency pore over stacks of evaluation research, seeking knowledge and insights that can help them design the new initiative. The evaluations provide them with lots of information about the effects of particular programs on particular people in particular settings. And having used random assignment to measure program effects, or impacts, the studies also afford considerable confidence in the reported results.

    Yet the research evidence is not as useful to the program designers as they had hoped. For all the studies’ rigor, many of them are not...

  7. Chapter 3 Constructing Instrumental Variables from Experimental Data to Explore How Treatments Produce Effects
    (pp. 75-114)
    Lisa A. Gennetian, Pamela A. Morris, Johannes M. Bos and Howard S. Bloom

    Arandom-assignment study can provide the most compelling evidence possible about how an intervention—be it social, economic, legal, or medical—affects the people to whom it is targeted. Randomization entails using a lotterylike process to assign each eligible sample member either to a group that is offered the intervention or to a group that is not. This process ensures that the two groups are the same in every way (in statistical expectation), except that one group is assigned to the intervention and the other is not. Any statistically significant differences between the two groups that are subsequently observed can be...

  8. Chapter 4 Randomizing Groups to Evaluate Place-Based Programs
    (pp. 115-172)
    Howard S. Bloom

    Social interventions such as community improvement programs, school reforms, and employer-based efforts to retain workers, whose aim is to change whole communities or organizations, are often called place-based initiatives. Because such programs are designed to affect the behavior of groups of interrelated people rather than individuals, it is generally not feasible to measure their effectiveness in an experiment that randomly assigns individuals to the program or to a control group. By randomizing at the level of groups such as neighborhoods, schools, or companies—also called clusters—researchers can still reap most of the methodological benefits of random assignment.

    Perhaps the...

  9. Chapter 5 Using Experiments to Assess Nonexperimental Comparison-Group Methods for Measuring Program Effects
    (pp. 173-236)
    Howard S. Bloom, Charles Michalopoulos and Carolyn J. Hill

    The past three decades have seen an explosion in the number of social program evaluations funded by government and nonprofit organizations. These evaluations span a wide range of policy areas, including education, employment, welfare, health, criminal justice, housing, transportation, and the environment. Properly evaluating a social program requires answering three fundamental questions: How was the program implemented? What were its effects? How did its effects compare with its costs?

    Perhaps the hardest part of the evaluation process is obtaining credible estimates of a program’s impacts. The impacts of a program are defined as the changes experienced by people exposed to...

  10. Index
    (pp. 237-252)