Skip to Main Content
Have library access? Log in through your library
Fighting for Reliable Evidence

Fighting for Reliable Evidence

Judith M. Gueron
Howard Rolston
Copyright Date: 2013
Published by: Russell Sage Foundation
Pages: 596
  • Cite this Item
  • Book Info
    Fighting for Reliable Evidence
    Book Description:

    Once primarily used in medical clinical trials, random assignment experimentation is now accepted among social scientists across a broad range of disciplines. The technique has been used in social experiments to evaluate a variety of programs, from microfinance and welfare reform to housing vouchers and teaching methods. How did randomized experiments move beyond medicine and into the social sciences, and can they be used effectively to evaluate complex social problems?Fighting for Reliable Evidenceprovides an absorbing historical account of the characters and controversies that have propelled the wider use of random assignment in social policy research over the past forty years.

    Drawing from their extensive experience evaluating welfare reform programs, noted scholar practitioners Judith M. Gueron and Howard Rolston portray randomized experiments as a vital research tool to assess the impact of social policy. In a random assignment experiment, participants are sorted into either a treatment group that participates in a particular program, or a control group that does not. Because the groups are randomly selected, they do not differ from one another systematically. Therefore any subsequent differences between the groups can be attributed to the influence of the program or policy. The theory is elegant and persuasive, but many scholars worry that such an experiment is too difficult or expensive to implement in the real world. Can a control group be truly insulated from the treatment policy? Would staffers comply with the random allocation of participants? Would the findings matter?

    Fighting for Reliable Evidencerecounts the experiments that helped answer these questions, starting with the income maintenance experiments and the Supported Work project in the 1960s and 1970s. Gueron and Rolston argue that a crucial turning point came during the 1980s, when Congress allowed states to experiment with welfare programs and foundations, states, and the federal government funded larger randomized trials to assess the impact of these reforms. As they trace these historical shifts, Gueron and Rolston discuss the ways that strategies for resolving theoretical and practical problems were developed, and they highlight the strict conditions required to execute a randomized experiment successfully. What emerges is a nuanced portrait of the potential and limitations of social experiments to advance empirical knowledge.

    Weaving history, data analysis and personal experience,Fighting for Reliable Evidenceoffers valuable lessons for researchers, policymakers, funders, and informed citizens interested in isolating the effect of policy initiatives. It is an essential primer on welfare policy, causal inference, and experimental designs.

    eISBN: 978-1-61044-813-0
    Subjects: Political Science

Table of Contents

Export Selected Citations Export to NoodleTools Export to RefWorks Export to EasyBib Export a RIS file (For EndNote, ProCite, Reference Manager, Zotero, Mendeley...) Export a Text file (For BibTex)
  1. Front Matter
    (pp. i-iv)
  2. Table of Contents
    (pp. v-vi)
    (pp. vii-viii)
    (pp. ix-x)
    (pp. xi-xiv)
    Judith M. Gueron and Howard Rolston
    (pp. xv-xviii)
  7. CHAPTER 1 Introduction: The Issue, the Method, and the Story in Brief
    (pp. 1-21)

    How can we know whether social programs do what they are designed to do? Can that question even be answered convincingly? A simple example illustrates the problem:

    The governor of a large midwestern state is under pressure. Welfare rolls are rising. Legislators are pushing for action. Armed with a report from a blue-ribbon panel, she recommends a new program with a catchy name: WoW, for Working over Welfare. A year later the program is up and running, the rolls are down, and the press declares a winner. Basking in the glow, she runs for national office.

    But did WoW actually...

  8. CHAPTER 2 They Said It Couldn’t Be Done: The National Supported Work Demonstration
    (pp. 22-65)

    The Supported Work project started out as a small local program offering subsidized jobs to two hard-to-employ groups (former prisoners and former addicts) in a structured environment, followed by assistance in finding an unsubsidized job. The national Supported Work demonstration, the subject of this chapter, built that concept into a successful random assignment experiment that tested the program’s effectiveness for the two original target groups and two additional ones: young high school dropouts and long-term welfare mothers. Eli Ginzberg, the chairman of MDRC’s board of directors, as quoted in its February 1975 press release announcing the demonstration, described it thus:...

  9. CHAPTER 3 Bridge to the 1980s: Implementing Random Assignment in Work Incentive Program Offices and Testing a Job Guarantee
    (pp. 66-87)

    This chapter tells the story of two initiatives that helped MDRC not only survive as an institution in the dramatically different policy and policy-research environment of the Reagan era but also continue using random assignment to learn more about program evaluation generally and the work-welfare area in particular. The first of these initiatives was the Work Incentive (WIN) Laboratory Project, notably the two job-search programs implemented in the Louisville, Kentucky, WIN program office. In many ways less ambitious than Supported Work, WIN Labs took an important next step by showing that random assignment could be implemented in regular government offices...

  10. CHAPTER 4 Inventing a New Paradigm: The States as Laboratories
    (pp. 88-140)

    Political transitions can sound portentous but may bring little change. For our story, however, Ronald Reagan’s election in 1980 was a true revolution, marking a dramatic turning point in welfare policy, the role of the states, and the nature and origin of welfare research. Support for a welfare entitlement and for voluntary programs such as Supported Work yielded to the harsher language of mandates and obligations. Policy initiative shifted from Washington to the more varied and less predictable terrain of the states. Funds for demonstrations evaporated, and social science researchers came to be viewed with suspicion, as advocates for the...

  11. CHAPTER 5 The Reagan Revolution and Research Capacity Within the Department of Health and Human Services: From Near Destruction to New Growth
    (pp. 141-180)

    When Ronald Reagan won the presidency in 1980, the Department of Health and Human Services (HHS) had already played a substantial role in what was then the relatively brief history of social experiments. Although the first such experiments were initiated by the Office of Economic Opportunity, through the 1970s HHS had initiated and supervised the operation of several negative-income-tax experiments, participated in the Supported Work demonstration, and been responsible for what is arguably the most policy-influential single social experiment ever conducted—the RAND Health Insurance Experiment.² The HHS expertise that developed from this experience was housed in the Office of...

  12. CHAPTER 6 The Work/Welfare Demonstration: Lessons About Programs, Research, and Dissemination
    (pp. 181-216)

    The Work/Welfare demonstration was the major turning point in our story. If Supported Work and the WIN Labs put a toe in the real world, this project was an immersion. At the start, we at MDRC fretted over all of the prerequisites to success: recruiting the states, getting the money, gaining federal cooperation, implementing random assignment, obtaining quality data, sustaining the project long enough to produce meaningful results, and completing the studies within budget. Once those challenges were overcome, we moved on to the payoff of distilling and sharing the lessons.

    The Work/Welfare demonstration ended up being a random assignment...

  13. CHAPTER 7 Waiver Evaluations: How Random Assignment Evaluation Came to Be the Standard for Approval
    (pp. 217-261)

    Throughout the first Reagan administration, the Department of Health and Human Services (HHS) applied the approval standard described in chapter 5 to section 1115 waiver applications: it approved waiver requests from states whose policy proposals were consistent with administration policy and denied those from states that wished to try anything different. Evaluation requirements were minimal, which was plainly permissible as a legal matter, given the simplicity of the standard and the broad secretarial discretion accompanying it. In this chapter I recount how that approval standard became increasingly untenable, as the federal stance on welfare reform shifted to the more federalist...

  14. CHAPTER 8 Expanding the Agenda: The Role of Basic Education
    (pp. 262-310)

    We have seen that real-world experiments proved feasible and credible; that most welfare-to-work programs were successful; that politicians and public officials listened and acted; that states could be willing partners; that a major foundation sparked these changes; and that several government agencies endorsed experiments as uniquely reliable. If welfare research had gone no further, it would already be unusual in its influence and in the use of multiple, large-scale experiments to determine the potential of a reform strategy.

    But the story did not end there. What makes the saga exceptional is that the momentum was sustained and resulted in a...

  15. CHAPTER 9 The JOBS Evaluation: Cumulative Evidence on Basic Education Versus Work First
    (pp. 311-353)

    Along with the Greater Avenues for Independence program (GAIN) evaluation (later, California’s JOBS program), the Job Opportunities and Basic Skills Training (JOBS) evaluation would profoundly alter the views of policy makers, program operators, and researchers about the effectiveness of alternative strategies to move welfare parents into the workforce.² Within a decade, the perspective underlying the JOBS program would be substantially undermined, and welfare-to-work programs would be very different from their predecessors under JOBS. Work programs under Aid to Families with Dependent Children (AFDC) and, later, Temporary Assistance for Needy Families (TANF) would be focused much less on basic education and...

  16. CHAPTER 10 Expanding the Agenda II: Three Experiments to Make Work Pay
    (pp. 354-387)

    With the welfare debate shifting rapidly toward the right in the early 1990s, work-for-your-grant (workfare) and related policies that had been denigrated as slavefare ten years earlier looked benign next to calls to end support. The tipping point came when the Democratic presidential candidate, Bill Clinton, made “ending welfare as we know it” and “making work pay” central parts of his platform, and the shift culminated in the 1996 law ending Aid for Families with Dependent Children (AFDC).

    During these years, as state governors proposed various combinations of carrots and sticks to influence a host of behaviors,² we at MDRC...

  17. CHAPTER 11 The End of Our Story: Welfare Reform Recedes from the Spotlight of Random Assignment Program Evaluation
    (pp. 388-422)

    Both Judy and I have heard people say, even experts very familiar with the welfare evaluation field, that the rigor of the large body of welfare experiments was a result of the federal government’s use of its 1115 waiver demonstration authority to require an experimental evaluation in exchange for programmatic flexibility (the quid pro quo). This explanation is appealing in its simplicity, but it is not how the random assignment or the waiver story actually developed. Since the waiver authority was never used to enforce the quid pro quo until 1987, and not used systematically for that purpose until 1992,...

  18. CHAPTER 12 Conclusion and Lessons
    (pp. 423-454)

    In the preceding chapters, we tell of our long struggle to show that a relatively untried technique—large-scale, random assignment experiments (the social policy analogue to clinical trials in medicine)—could be used to answer important questions about complex social programs. We also show that the results were considered singularly credible and, in part because of that, were judged by some to have an outsized impact on policy and practice.

    Our story is about one policy area—welfare reform—as viewed through the lens of two institutions that worked independently and in interaction with others to promote this method and...

  19. CODA Random Assignment Takes Center Stage
    (pp. 455-472)

    This book describes the struggle to promote randomized experiments as a campaign waged over many years, mostly by people outside the normal academic reward system. In recent years, this has changed, both in the United States and internationally, prompting me to ask a number of people (see the appendix) close to the past conflict or current transformation: How would you characterize academic and government support for experiments forty years ago? Has this changed? If so, why? Although by no means a systematic description of this evolution, their responses provide insight on the context within which our story unfolded.

    Larry Orr,...

    (pp. 473-478)
    (pp. 479-480)
  22. NOTES
    (pp. 481-536)
    (pp. 537-538)
    (pp. 539-552)
  25. INDEX
    (pp. 553-576)