Skip to Main Content
Have library access? Log in through your library
Readings in Evaluation Research, 2ed

Readings in Evaluation Research, 2ed

EDITED BY Francis G. Caro
Copyright Date: 1977
Published by: Russell Sage Foundation
Pages: 448
https://www.jstor.org/stable/10.7758/9781610441094
  • Cite this Item
  • Book Info
    Readings in Evaluation Research, 2ed
    Book Description:

    Affords a comprehensive overview of evaluative research, answering questions regarding the adequacy of organized programs in health, justice, education, employment, and welfare. Included are general statements about evaluative research, discussing the nature of the evaluative task, the role of evaluative research in programs for change, and appropriate methodological strategies. In this revised and expanded collection of readings, which includes more case materials and more illustrations of completed evaluations than the first edition, the editor presents a variety of viewpoints and a broad range of materials for the social planner, administrator, and social scientist.

    eISBN: 978-1-61044-109-4
    Subjects: Sociology

Table of Contents

Export Selected Citations Export to NoodleTools Export to RefWorks Export to EasyBib Export a RIS file (For EndNote, ProCite, Reference Manager, Zotero, Mendeley...) Export a Text file (For BibTex)
  1. Front Matter
    (pp. i-iv)
  2. Table of Contents
    (pp. v-viii)
  3. FOREWORD
    (pp. ix-x)
    HOWARD E. FREEMAN

    The evaluation research field has seen remarkable growth in the brief six years since the publication of the first edition of Francis Caro’sReadings in Evaluation Research.Growth has occurred within the field itself, in the amount of governmental and private sector funding for evaluation studies, in the number of investigators who identify themselves as evaluation researchers, and in the attention paid to evaluation research in graduate academic programs. The growth is evident in other ways as well: in efforts to improve the methodology of evaluation research, in attempts to solve the practical and political problems that often plague the...

  4. PREFACE
    (pp. xi-2)
    FRANCIS G. CARO
  5. EVALUATION RESEARCH: AN OVERVIEW
    (pp. 3-38)

    Evaluation in recent years has come to be recognized as a distinct and important dimension of social programming. Not only has there been a growth in public concern about social problems, but there has been an increasing recognition of a need for sophistication in the design and administration of social programs. The time when goodwill, dedication, and hard work were thought to be sufficient in ameliorating social problems is behind us. Failure of highly publicized, attractive programs to fulfill their promise has helped to create demand for careful program review. Recognition that public and private expenditures for social programs are...

  6. PART I: BASIC ISSUES:: PROGRAM DEVELOPMENT AND SCIENTIFIC INQUIRY

    • 1. Prospects and Possibilities: The New Deal and the New Social Research
      (pp. 40-47)
      A. Stephen Stephan

      Mankind in a test-tube is the hope and aim of social science.

      Students of human behavior have long envied the chemists and physicists who are releasing the secrets nature through experimentation and laboratory procedure. The exacting methods of the laboratory have been responsible for the phenomenal advance of the physical sciences. The gap between the accumulated knowledge of the physical sciences and the social sciences is largely explained by the difference in the exact methods of the former and the floundering methods the latter. Man knows more about the atom than he knows about himself.

      The promise of a more...

    • 2. Evaluating Educational Programs
      (pp. 48-53)
      Edward A. Suchman

      By and large, researchers have been reluctant to undertake evaluation studies. The basis for such resistance lies mainly in the general inadequacy of many of such studies judged by scientific standards. While this poor reputation may be justified from past experience,¹ the shortcomings are not inherent in the conduct of evaluation studies. The purpose of this paper will be to formulate some of the basic issues involved in viewing evaluation as research and to point out some of the ways in which such studies can be improved.

      The key conceptual elements in a definition of evaluation from a methodological point...

    • 3. Program Evaluation Models and the Implementation of Research Findings
      (pp. 54-64)
      Herbert C. Schulberg and Frank Baker

      A source of great dismay to both the researcher and the clinician is the difficulty encountered in trying to apply the findings of a research project. This consternation is particularly acute in the research specialty of program evaluation, since both the program administrator and program evaluator undertake studies with the fullest and sincerest intention of utilizing the resulting data. The reasons for the gap between research and implementation are varied and considerable attention has been devoted in recent years to analysis of personal and organizational resistances to change. This paper restricts its focus to the issues specifically relevant to program...

  7. PART II: THE ORGANIZATIONAL CONTEXT:: ESTABLISHING AND MAINTAINING THE EVALUATION RESEARCH ROLE

    • [PART II Introduction]
      (pp. 65-66)

      The task of evaluation research is to determine the effects of programs conducted in an organizational setting. Often the results are used to help a sponsor decide whether an organization deserves continued support, or to assist the administering organization itself in modifying its programming. Effective working relationships with those who administer programs, therefore, are critical for evaluative researchers.

      In devising an evaluative research plan, the social researcher and his client may do well to consider various evaluation alternatives as investment strategies. Downs urges that the cost of research be considered in the light of the economic benefits the research might...

    • 4. Some Thoughts on Giving People Economic Advice
      (pp. 67-72)
      Anthony Downs

      Surprisingly, economists seem to have developed few theories about how to give other people advice effectively. True, there is a vast literature on how to make decisions. There are also extensive writings on which types of advice from economists can be considered purely scientific, and which must also be considered partly ethical. Finally, there are tons of books and articles concerning the substantive issues which advisors are likely to grapple with.

      Nevertheless, there is a significant gap in both empirical data and theory concerning the kinds of relationships likely to develop between an advisor and the decision-makers who seek his...

    • 5. Organizational Strains in the Researcher-Practitioner Relationship
      (pp. 73-93)
      Hyman Rodman and Ralph Kolodny

      Social science researchers have, to an increasing extent, been moving into clinical settings, such as mental hospitals, general hospitals, child guidance clinics, and social work agencies, and into other professional settings, such as schools and courts. It is well known that problems arise when a social science researcher enters a clinical agency or some other professional setting. What we are interested in exploring is whether there are similarities in the problems faced by researchers and practitioners in these professional agencies, and whether certain of these problems stem from the organizational structure of the professional agency. We shall deal primarily with...

    • 6. The Politics of Evaluation in Higher Education
      (pp. 94-101)
      Ernest R. House

      Recently, in connection with evaluations conducted in higher education, I have found myself in shouting matches with vice-chancellors, have had my work censored, and have heard lawsuits mentioned. I have been in so much political trouble that I have begun seeing myself as the “Kojak” of the evaluation world. As I look around, I see my colleagues in similar difficulties. I think there are common causes for these troubles, causes that lie within the structure of colleges and universities themselves.

      Much program evaluation in higher education revolves around special projects, often funded by outside agencies which demand an evaluation [2]....

    • 7. Program Management and the Federal Evaluator
      (pp. 102-114)
      Pamela Horst, Joe N. Nay, John W. Scanlon and Joseph S. Wholey

      In 1969, The Urban Institute completed an extensive study of federal evaluation and concluded that, “The most impressive finding about the evaluation of social programs in the federal government is that substantial work in this field is almost nonexistent.”¹ A limited resurvey of the field in 1972 revealed a quite different picture: funds committed to evaluation had mushroomed, many studies had been completed, and the use of large-scale social experimentation was increasing.²

      This growth in evaluation has contributed information—often imperfect, sometimes incorrect—to today’s arguments about the direction, method, and purpose of social programs. Without evaluation, many arguments would...

    • 8. Between the Cup and the Lip
      (pp. 115-124)
      Carol H. Weiss

      Evaluation research calls for research skills plus a diverse array of talents to facilitate the use of those skills in a basically inhospitable setting. Recognition of this fact came as a result of study by Columbia University’s Bureau of Applied Social Research of 10 applied research projects. Seven of the projects were funded by NIMH, three by other federal agencies. While not all of them were specifically designed as program evaluations, all but one had an immediate or prospective evaluative orientation.

      Selection of the NIMH projects was based largely on two criteria: first , that the project had paid serious...

    • 9. The Need for Research on the Communication of Research Results
      (pp. 125-134)
      Floyd Mann and Rensis Likert

      Research on problems of human relations differs from research in most other fields of science in a very important respect. In most fields of science it is not necessary for administrators or executives to have a comprehensive understanding of the research in order to utilize the results. All that has to be known is that the research has yielded a better method or a better product. Approval to substitute the new for the old can then be given. But in the field of human relations, effective use of the research findings cannot be obtained merely by an executive issuing an...

  8. PART III: METHODOLOGICAL ISSUES:: MEASUREMENT AND DESIGN

    • [PART III Introduction]
      (pp. 135-136)

      Use of the scientific method in obtaining information distinguishes evaluative research from general evaluation processes. The basic methodological principles that apply to evaluative research are the same as those used in traditional scientific inquiry. The situations with which the evaluative researcher is confronted, however, are such that the methods of social research must be applied in specialized fashion. The papers in this section are addressed to the methodological issues posed by evaluative research.

      Greenberg offers a thorough and balanced presentation of the essentials of evaluation methodology, with emphasis on application to public health problems. Particularly valuable is his treatment of...

    • 10. Evaluation of Social Programs
      (pp. 137-158)
      B. G. Greenberg

      Governmental programs which provide social and educational services to the public are generally costly in terms of money and manpower. Public administrators of such programs have the responsibility to account for their accomplishments not only because of the scarcity of these resources but the sometimes dreadful consequences which can result to the people from poorly administered services. In health, educational, and welfare activities the damage caused by inadequate service is often irreversible. It may be too late for the victims of poorly designed or inefficiently administered programs to have deficiencies corrected at a subsequent date. Public accountability requires advance planning...

    • 11. Evaluative Studies of Institutions for Delinquents: Implications for Research and Social Policy
      (pp. 159-171)
      Paul Lerman

      Evaluative research is usually undertaken for the purpose of gathering evidence of a program’s success in achieving its avowed goals.¹ This approach can be questioned, however, unless a more basic question has first been answered in the affirmative: Is there any empirical evidence that the program under consideration is more likely to be associated with success than with failure? It is not sufficient merely to assume that assessing success is the relevant evaluative problem. One must be willing to face the possibility that the program is associated with high rates of failure. Instead of the success of a program, it...

    • 12. Reforms as Experiments
      (pp. 172-204)
      Donald T. Campbell

      The United States and other modern nations should be ready for an experimental approach to social reform, an approach in which we tryout new programs designed to cure specific social problems, in which we learn whether or not these programs are effective, and in which we retain, imitate, modify, or discard them on the basis of apparent effectiveness on the multiple imperfect criteria available. Our readiness for this stage is indicated by the inclusion of specific provisions for program evaluation in the first wave of the “Great Society” legislation, and by the current congressional proposals for establishing “social indicators” and...

    • 13. Research in Large-Scale Intervention Programs
      (pp. 205-220)
      Howard E. Freeman and Clarence C. Sherwood

      Dissatisfaction with the social order and zealous efforts at community change have characterized the personal and academic lives of social scientists since their emergence as an identifiable group on the American scene.¹ In many ways, of course, the various disciplines and the persons that hold membership in them have changed markedly over the last several decades: the influence of visionary clergymen, guilt-ridden do-gooders, and political radicals—dedicated to projecting their own humanitarian views in the guise of scientific inquiry—has pretty well diminished.²

      But the social scientist has expanded his role in the modification of community life and in the...

    • 14. Toward Avoiding the Goal Trap in Evaluation Research
      (pp. 221-238)
      Irwin Deutscher

      It is the purpose of this paper to devise means which may be helpful in implementing two values. If one does not concur in either of them, much of the paper will make no sense. If one does concur in both values, some of the paper may make some sense. First, it is desirable to determine the consequences of deliberate efforts to alter ongoing social processes—education, health, welfare, or whatever—and to do so in as detached a manner as is possible. A corollary to this value is that it is desirable to understand the new processes which lead...

    • 15. Boobytraps and Pitfalls in the Evaluation of Social Action Programs
      (pp. 239-248)
      Peter H. Rossi

      If one were to measure success by the popularity of evaluation research, then empirical social research has certainly arrived. Perhaps, the best example of this popularity lies in the legislation authorizing the present War on Poverty in which the agencies involved are specifically directed to set aside funds for evaluation research. Other ameliorative programs may not give as much formal recognition to such activity, but nevertheless seek social researchers to add to their staffs for this purpose or attempt to get social research centers to provide evaluations of their programs.

      There are other measures of success besides popularity. If one...

  9. PART IV: CASE MATERIALS

    • [PART IV Introduction]
      (pp. 249-252)

      This final section consisting of reports on specific evaluation studies is intended to complement the general papers in previous sections. The case materials illustrate the application of general principles and the field experiences of researchers in conducting evaluation studies. Some of the case materials deal with the circumstances surrounding evaluation projects. These papers deal with the rationale for use of particular evaluation strategies in the situations described. The political and social policy context of evaluation projects is also illustrated in some of these cases. Other papers in this section are reports of completed evaluations. Studies are included of interventions in...

    • 16. The Evaluation of Broad-Aim Programs: A Cautionary Case and a Moral
      (pp. 253-262)
      Robert S. Weiss and Martin Rein

      There is an approach to the evaluation of programs of social action which seems so sensible that it has been accepted without question in many quarters. The underlying assumption of the approach is that action-programs are designed to achieve specific ends, and that their success can be measured by the extent to which these ends were reached. The approach leads often to study-design in experimental form, in which there is identification of the aspects of the situation or target population which are to be changed, the measurement of their state before introduction of the program, and the measurement of their...

    • 17. The Evolution of an Evaluation: Methodological Problems in Programs for School-Age Mothers
      (pp. 263-276)
      Lorraine V. Klerman, James F. Jekel, John B. Currie, Ira W. Gabrielson and Philip M. Sarrel

      In this time of technological progress and relative prosperity, the American public has come to expect much from the fields of health, education, and social assistance; but these expectations are continually disappointed by persistent ill health, low levels of education, and poverty among sizeable portions of the population. The traditional methods of delivering services in these fields are increasingly being questioned, and evidence of success is being sought to justify continued funding. Although evaluation has always been considered an integral part of administration, a lack of funds and skilled personnel usually has prevented studies of effectiveness and efficiency from being...

    • 18. Experimental Methodology and Innovative Social Programming
      (pp. 277-285)
      Francis G. Caro

      Researchers with disciplinary roots in the behavioral sciences characteristically regard the formal experiment as the preferred model for the evaluation of innovative social programming. (See, for example, Campbell, 1969 and Rivlin, 1971.) So that sound inferences can be made regarding the effectiveness of an intervention, it is recommended that those receiving the innovative treatment be compared with an equivalent group that does not receive the innovative treatment. Random assignment of potential recipients to experimental and control conditions is considered an ideal procedure for assuring comparability of groups. The strategy calls for repeated measurement of those conditions in which change is...

    • 19. The External Politics of the National Income Transfer Experiment
      (pp. 286-295)
      Peter H. Rossi and Katharine C. Lyall

      All large-scale, policy oriented research projects are conducted within political context, and all research operations create their own internal political systems. The context for a research project includes the organizations within which the project is embedded, the relations with sponsors and sources of funds, audiences that are willing or not willing to learn of results, and sometimes even the subjects of the research or their organized representatives. For large-scale policy oriented research sponsored by a controversial government agency, these elements of context impinge with particular force. The National Income Transfer (NIT) project was no exception: There were several instances when...

    • 20. Responding to Skid Row Alcoholism: Self-Defeating Arrangements in an Innovative Treatment Program
      (pp. 296-309)
      Lincoln J. Fry and Jon Miller

      Based upon twenty-seven months of participant observation in a skid row mission, this study explores some of the factors that contributed to the ineffectiveness and ultimate failure of what began. as a highly promising alcoholism rehabilitation program. The program was very generously funded from state and local resources; it was technically and philosophically innovative; it was highly professionalized and staffed by young, bright, eager practitioners. Further, it was located in the treatment quarters of an established, highly regarded religious organization that could add stability and experience to this impressive array of financial and professional resources. Despite these advantages the experiment...

    • 21. An Experiment on the Social Effects of Good Housing
      (pp. 310-322)
      F. Stuart Chapin

      Is the condition of a slum family improved by rehousing in a model public housing project? An affirmative answer to this question is assumed as the justification for the expenditure of millions of dollars. Is there any proof of this assumption aside from common sense expectation?

      This study is an effort to measure the effects of good housing upon former slum families rehoused in Sumner Field Homes of Minneapolis, originally a project of the Housing Division of the PWA, and since 1937, under the management of the USHA.

      The most interesting findings of this study are: (1) no significant change...

    • 22. The Kansas City Preventive Patrol Experiment: A Summary
      (pp. 323-342)
      George L. Kelling, Tony Pate, Duane Dieckman and Charles E. Brown

      Ever since the creation of a patrolling force in 13th century Hangchow, preventive patrol by uniformed personnel has been a primary function of policing. In 20th century America, about $2 billion is spent each year for the maintenance and operation of uniformed and often superbly equipped patrol forces. Police themselves, the general public, and elected officials have always believed that the presence or potential presence of police officers on patrol severely inhibits criminal activity.

      One of the principal police spokesmen for this view was the late O. W. Wilson, former chief of the Chicago Police Department and a prominent academic...

    • 23. The Impact of Head Start: Executive Summary
      (pp. 343-347)
      Victor Cicarelli

      This report presents the results of a study on the impact of Head Start carried out for the Office of Economic Opportunity from June 1968 through May 1969 by Westinghouse Learning Corporation and Ohio University.

      The study attempted in a relatively short period of time to provide an answer to a limited question concerning Head Start’s impact; namely: Taking the program as a whole as it has operated to date, to what degree has it had psychological and intellectual impact on children that has persisted into the primary grades?

      The very real limitation of our study should be established at...

    • 24. Head Start: Comments on the Criticisms
      (pp. 348-354)
      John W. Evans

      Since its release, the Westinghouse study has occasioned considerable, even bitter, debate. If we are to understand the heat that has spiced this controversy, in my opinion we must look less to the purely methodological issues in the exchange and more to the fact that the findings of the study have been difficult to accept. Head Start has been the showcase program of the war on poverty. As a bold new effort to prevent the numbing effects of poverty on small children, it elicited immediate national sympathy as well as the support and involvement of the education profession. It is...

    • 25. An Evaluation of the Effectiveness of the OEO Legal Services Program
      (pp. 355-371)
      Anthony Champagne

      In view of the current controversy over the legal services program (Pious, 1972), it is important to answer the question: What has been the impact of the OEO projects? The question will be answered by determining the degree of effectiveness of the program in meeting its goals of individual case handling, law reform, and community development.¹ In addition, evaluations of the effectiveness of legal services as determined by administrators of agencies and institutions relevant to poor people, poverty community group leaders, and the legal community will be discussed.

      Before the discussion, it should be pointed our that the term “effectiveness”...

    • 26. Champagne’s Assessment of Legal Services Programs: An Evaluation of an Evaluation
      (pp. 372-385)
      Richard A. Berk

      For the vast majority of public programs, no scientific evaluation is ever undertaken. With poorly defined goals, empirical measures of dubious quality, andad hominumcriteria for judging effectiveness, a program’s life expectancy typically depends on considerations marginally related to reasonably objective assessments of performance. It is now widely accepted that the community action program Lyndon Johnson’s War on Poverty, for example, was in trouble from its inception because of vague legislation, sloppy monitoring, and political exigencies (Moynihan, 1969; Marris and Rein, 1967; Clark and Hopkins, 1968). And while there is no such thing as a totally objective evaluation (it...

    • 27. A Reply to Berk
      (pp. 386-388)
      Anthony Champagne

      Berk provides a useful criticism evaluation research, though a number of his comments regarding the Auerbach-Kettelle data deserve further mention. Rather than make a lengthy response to the comments offered by Professor Berk, this reply will briefly mention what I believe are Berk’s key points.

      Few would doubt the importance obtaining a “consumer perspective” the legal services program.¹ If one is to examine the effectiveness of a program such as legal services, it would be ideal to know what the clients of the legal services program feel. Unfortunately, Professor Berk chose not to scan the several volumes of materials on...

    • 28. A National Evaluation of Community Services and the Quality of Life in American New Towns
      (pp. 389-403)
      Raymond J. Burby III, Shirley F. Weiss and Robert B. Zehner

      Population growth, city problems, and expanding expectations for a better life and a better environment have focused national attention on new communities as a potentially better form urban development. Depending on definitions and standards, estimates the number of new communities in America range between 50 and 250 communities which might ultimately house from five to 25 million persons. Regardless of the exact number, this clearly represents an unprecedented level of private commitment to new community development. The private sector’s expanding involvement community building has provided public agencies with an opportunity and a challenge to match excellence in land development with...

    • 29. Acute Myocardial Infarction: Home and Hospital Treatment
      (pp. 404-413)
      H. G. Mather, N. G. Pearson, K. L. Q. Read, D. B. Shaw, G. R. Steed, M. G. Thorne, S. Jones, C. J. Guerrier, C. D. Eraut, P. M. McHugh, N. R. Chowdhury, M. H. Jafary and T. J. Wallace

      This is a preliminary report of a cooperative study of 1,203 episodes of acute myocardial infarction in men under 70 years in four centres in the south west of England. The mortality at 28 days was 15%. A comparison is made between home care by the family doctor and hospital treatment initially in an intensive care unit: 343 cases were allocated at random. The randomized groups do not differ significantly in composition with respect to age; past history of angina, infarction, or hypertension; or hypotension when first examined. The mortality rates of the random groups are similar for home and...

    • 30. Determining the Social Effects of a Legal Reform: The British “Breathalyser” Crackdown of 1967
      (pp. 414-428)
      H. Laurence Ross, Donald T. Campbell and Gene V. Glass

      The social effects of a legal reform are examined in this paper utilizing the Interrupted Time-Series research design, a method of analysis that has broad potential use in studies of legal change more generally. A previous demonstration of the applicability of this design to the sociology of law concerned the Connecticut crackdown on speeders (see Campbell and Ross, 1968; Glass, 1968). In that study, the substantive findings were that the crackdown had little effect on the highway death rate, and that it introduced certain unexpected and undesirable changes into the legal process in Connecticut. The present study concerns a similar...

  10. NAME INDEX
    (pp. 429-434)
  11. SUBJECT INDEX
    (pp. 435-436)