Running Randomized Evaluations

Running Randomized Evaluations: A Practical Guide

Copyright Date: 2013
Edition: STU - Student edition
Pages: 472
  • Cite this Item
  • Book Info
    Running Randomized Evaluations
    Book Description:

    This book provides a comprehensive yet accessible guide to running randomized impact evaluations of social programs. Drawing on the experience of researchers at the Abdul Latif Jameel Poverty Action Lab, which has run hundreds of such evaluations in dozens of countries throughout the world, it offers practical insights on how to use this powerful technique, especially in resource-poor environments.

    This step-by-step guide explains why and when randomized evaluations are useful, in what situations they should be used, and how to prioritize different evaluation opportunities. It shows how to design and analyze studies that answer important questions while respecting the constraints of those working on and benefiting from the program being evaluated. The book gives concrete tips on issues such as improving the quality of a study despite tight budget constraints, and demonstrates how the results of randomized impact evaluations can inform policy.

    With its self-contained modules, this one-of-a-kind guide is easy to navigate. It also includes invaluable references and a checklist of the common pitfalls to avoid.

    Provides the most up-to-date guide to running randomized evaluations of social programs, especially in developing countries Offers practical tips on how to complete high-quality studies in even the most challenging environments Self-contained modules allow for easy reference and flexible teaching and learning Comprehensive yet nontechnical

    eISBN: 978-1-4008-4844-7
    Subjects: Business, Political Science

Table of Contents

  1. Front Matter
    (pp. i-iv)
  2. Table of Contents
    (pp. v-vi)
    (pp. vii-viii)
    (pp. ix-xii)
  5. 1 The Experimental Approach
    (pp. 1-23)

    In 1994 I went with Michael Kremer to visit the family he had lived with for a year in rural Kenya.¹ We met up with many of Michael’s old friends, including Paul Lipeyah, who told us of the work he was doing with International Child Support (ICS) Africa, a nongovernmental organization (NGO) helping government schools in Busia, a neighboring district in Kenya’s Western Province. Paul asked us what advice we might offer for improving the effectiveness of ICS programs. Could Michael help evaluate what they were doing? Michael suggested randomized evaluation: if ICS wanted to understand the impact of their...

  6. 2 Why Randomize?
    (pp. 24-65)

    Causal impact is the difference in outcomes that is caused by the program. In other words, to estimate the impact of a program, we need to examine how the people who participated in the program fared compared to how they would have fared if they had not participated in the program. This hypothetical condition is called the counterfactual. However, we never observe the counterfactual directly. We observe only what happens with the program, not what would have happened in the absence of the program, and so we have to make an inference about the counterfactual. This is the fundamental problem...

  7. 3 Asking the Right Questions
    (pp. 66-97)

    In the previous chapter we saw that performing a randomized evaluation confers a number of advantages for generating the information that we need for evidence-based policy. But undertaking a good randomized evaluation is hard and can be expensive. We should undertake one only when the benefits from the lessons we learn are likely to outweigh the costs. We should not conduct a randomized evaluation every time we implement a program. Nor do we need a randomized evaluation every time we have a question about whether and how a program is working. Many important questions can be answered by other types...

  8. 4 Randomizing
    (pp. 98-179)

    In order to evaluate the impact of a program or policy, our randomly selected treatment group must have more exposure to the program than the comparison group. We can control three aspects of the program or policy to create this differential exposure:

    1. Access: We can choose which people will be offered access to the program.

    2. Timing of access: We can choose when to provide access to the program.

    3. Encouragement: We can choose which people will be given encouragement to participate in the program.

    Because we control these three aspects of the program, these are also the three...

  9. 5 Outcomes and Instruments
    (pp. 180-240)

    Specifying good outcomes and the indicators we will use to measure them requires a deep understanding of the program being developed, the objectives of those implementing the program, and potential pathways through which the program or policy can impact lives, both positively and negatively. Some of the terms we use are defined with examples in Table 5.1. A theory-of-change framework is a useful tool to use in systematically thinking through potential pathways to an impact and is best developed in close cooperation between those implementing and those evaluating a program. This theory of change is likely to lead us to...

  10. 6 Statistical Power
    (pp. 241-297)

    We have completed our evaluation of a program designed to improve test scores in a poor community in India. We find that the average score on the test we administer is 44 percent in the treatment group and 38 percent in the comparison group. Can we be reasonably confident that this difference of 6 percentage points is due to the program, or could it be due to chance? We perform a statistical test designed to check whether the difference we observe could be due to chance. We find that the difference between the treatment and comparison groups is not statistically...

  11. 7 Threats
    (pp. 298-323)

    Some of the people in the treatment group may never be treated. For example, some students assigned to a training program may never attend the training. Some parents whose children are assigned to receive deworming drugs may not give their consent. Or impassable roads in the rainy season may keep a program from delivering fertilizer to some farmers in time for planting. We can measure the extent of this partial compliance from process data collected throughout the implementation as we check records of how many people attend the training program or how many parents do not give permission for deworming...

  12. 8 Analysis
    (pp. 324-385)

    Before we start any analysis, we need to prepare (or “clean”) our data and make sure we have a good understanding of their properties. This usually involves taking the following steps:¹

    Correcting obvious errors in the data

    Checking for outliers

    Calculating attrition rates

    Calculating compliance rates

    Plotting and describing the data

    It is good practice to look at our data and make sure that we have corrected obvious errors in the data (sometimes called “cleaning the data”). Errors can occur when a respondent misunderstands the question, when an enumerator fills in the questionnaire incorrectly, or when data are entered incorrectly....

  13. 9 Drawing Policy Lessons
    (pp. 386-420)

    One benefit of the randomized evaluation methodology is that there are some basic criteria by which to judge whether a study is valid. This (nonexhaustive) checklist provides a summary of some of the most common mistakes made and refers the reader to the relevant sections of the book that discuss each issue in greater depth. We group the mistakes into those that are made at the design, implementation, and analysis stages.

    Often providing a program to one member of a community will have implications for other members of the community. If we provide information to a random sample of farmers,...

    (pp. 421-442)
    (pp. 443-452)
  16. INDEX
    (pp. 453-467)