There is no commonly accepted definition of the term “evaluation research.” As we use the term here, it is the systematic study of the operation of social-action, treatment, or intervention programs, and of their impact. Ideally, the goals of evaluation research are twofold: first, to supply information that allows policy-makers, planners, and professionals to make rational decisions about social-development and human-resource programs and thus to maximize the expenditure of economic and human resources; and, second, to add to knowledge available about social and interpersonal behavior and the social environment, and to explicate and refine the practice principles that underlie programming...

This study was developed, as stated in the Preface, as part of a broad program at Russell Sage Foundation directed toward the improvement of evaluation-research activities. During the early stages of the development of the overall program, staff at the Foundation spent considerable time engaged in discussions with persons involved in the administration and funding of evaluation-research activities in the various departments of the executive branch of the federal government. In addition, government officials in the Office of the President responsible for domestic programs and officials in the Office of Management and Budget were contacted. Their views on the current...

Despite the mounting interest in evaluation research, the expansion of federal funds for studies and the proliferation of organizations engaged in investigations, as indicated previously, little is known about the sources of support, the background of persons involved in the enterprise, the organizational arrangements which surround activities, the methodological techniques employed, or the dissemination of research results. The research enterprise is inextricably bound up with the way the complex federal human-resource establishment is structured and operates. Thus, the organizational arrangements in evaluation research and the nature of activities undertaken need to consider the relations between funding groups and awardees, as...

It is our contention, as discussed in Chapter 1, that evaluation-research efforts should contain procedures to measure process if the evaluation of a social-action program or intervention is to be comprehensive and have utility. Hyman and Wright (1967, p. 744) underscore this by stating, “Taking the work for the deed, an evaluator may try to observe the effects of nonexistent treatment, or a treatment very different from the one he thought was being examined.” A noteworthy case of this kind was made famous by the same two authors, who recount the tale of the evaluation of a propaganda campaign based...

The measurement of impact requires procedures that allow the evaluator (1) to document the extent to which the social-action program has or has not achieved its stated goals; (2) to attribute any effects or changes that are discovered to the implementation of the action program, ruling out rival hypotheses which might alternatively explain identified changes; (3) to delineate, if possible, the conditions or combination of conditions under which the program is most effective (i.e., those that yield maximum benefits and minimum costs); and (4) to delineate, if possible, any unanticipated consequences or side effects of the implementation of the program....

Even when self-reports of investigators on the procedures employed in evaluation research are the data base, there is great variation in design, sampling, measurement, and data-analysis procedures. Moreover, although many in the field assert that evaluations should be comprehensive (e.g., Suchman 1967 and Greenberg 1968), in many cases they are not.

The preceding two chapters discussed some of the basic procedures employed and ordered studies according to variables which we maintain reflect differential quality of evaluation research. We related the ordering of studies on these indicators of quality to a series of measures that describe the evaluation-research enterprise. Our analysis...

From the standpoint of research quality, there is an obvious solution for improving evaluation research: have all studies undertaken in academic research centers, by Ph.D. psychology professors and those with similar training and orientation, include a commitment that the research results must be published in refereed social science journals, provide funds only as grants and have them awarded on the basis of peer-review committee judgments, allow a time period of from three to five years for the planning and conduct of the research, have the grants monitored by federal officials with a high degree of social-science graduate training and with...