Expert Political Judgment

Expert Political Judgment: How Good Is It? How Can We Know?

Philip E. Tetlock
Copyright Date: 2005
Edition: STU - Student edition
Pages: 352
https://www.jstor.org/stable/j.ctt7spbt
  • Cite this Item
  • Book Info
    Expert Political Judgment
    Book Description:

    The intelligence failures surrounding the invasion of Iraq dramatically illustrate the necessity of developing standards for evaluating expert opinion. This book fills that need. Here, Philip E. Tetlock explores what constitutes good judgment in predicting future events, and looks at why experts are often wrong in their forecasts.

    Tetlock first discusses arguments about whether the world is too complex for people to find the tools to understand political phenomena, let alone predict the future. He evaluates predictions from experts in different fields, comparing them to predictions by well-informed laity or those based on simple extrapolation from current trends. He goes on to analyze which styles of thinking are more successful in forecasting. Classifying thinking styles using Isaiah Berlin's prototypes of the fox and the hedgehog, Tetlock contends that the fox--the thinker who knows many little things, draws from an eclectic array of traditions, and is better able to improvise in response to changing events--is more successful in predicting the future than the hedgehog, who knows one big thing, toils devotedly within one tradition, and imposes formulaic solutions on ill-defined problems. He notes a perversely inverse relationship between the best scientific indicators of good judgement and the qualities that the media most prizes in pundits--the single-minded determination required to prevail in ideological combat.

    Clearly written and impeccably researched, the book fills a huge void in the literature on evaluating expert opinion. It will appeal across many academic disciplines as well as to corporations seeking to develop standards for judging expert decision-making.

    eISBN: 978-1-4008-3031-2
    Subjects: Political Science, Psychology

Table of Contents

  1. Front Matter
    (pp. i-vi)
  2. Table of Contents
    (pp. vii-viii)
  3. Acknowledgments
    (pp. ix-x)
  4. Preface
    (pp. xi-xviii)
  5. CHAPTER 1 Quantifying the Unquantifiable
    (pp. 1-24)

    Every day, countless experts offer innumerable opinions in a dizzying array of forums. Cynics groan that expert communities seem ready at hand for virtually any issue in the political spotlight—communities from which governments or their critics can mobilize platoons of pundits to make prepackaged cases on a moment’s notice.

    Although there is nothing odd about experts playing prominent roles in debates, it is odd to keep score, to track expert performance against explicit benchmarks of accuracy and rigor. And that is what I have struggled to do in twenty years of research of soliciting and scoring experts’ judgments on...

  6. CHAPTER 2 The Ego-deflating Challenge of Radical Skepticism
    (pp. 25-66)

    It is commonplace to lament the sad state of political forecasting. Moreover, suspicions that the entire enterprise is intellectually bankrupt have only been fortified by the most recent forecasting fiasco: the unanimous declaration by quantitative modelers of presidential elections at the American Political Science Association in August 2000 that we could ignore the frantic rhetorical posturing of the next few months. Election campaigns are tales full of sound and fury but of no significance because of the offsetting effects of each side’s propaganda broadsides. The die had been cast: Gore would defeat Bush by decisive, even landslide, margins.¹

    We revisit...

  7. CHAPTER 3 Knowing the Limits of One’s Knowledge: FOXES HAVE BETTER CALIBRATION AND DISCRIMINATION SCORES THAN HEDGEHOGS
    (pp. 67-120)

    Behavioral scientists often disagree over not only the facts but also over what is worth studying in the first place. What one school of thought dismisses as a minor anomaly of no conceivable interest to any gainfully employed grown-up, another school elevates to center stage. So it is here. The competing perspectives on good judgment in chapter 1—skepticism and meliorism—offer starkly different assessments of the wisdom of searching for correlates of forecasting skill. Skeptics argue that chapter 2 settled the debate: good judgment and good luck are roughly one and the same. Meliorists sigh that chapter 3 finally...

  8. CHAPTER 4 Honoring Reputational Bets: FOXES ARE BETTER BAYESIANS THAN HEDGEHOGS
    (pp. 121-143)

    Chapters 2 and 3 measured expert performance against correspondence benchmarks of good judgment. The test was “getting it right”: affixing realistic probabilities to possible futures. The spotlight shifts in chapters 4 and 5 to coherence and process benchmarks of good judgment. The focus is on “thinking the right way”: judging judgment on the logical soundness of how we go about drawing inferences, updating beliefs, and evaluating evidence. These alternative conceptions of good judgment are more complementary than competitive. It would be odd if people who think the right way failed to get more things right in the long run. Indeed,...

  9. CHAPTER 5 Contemplating Counterfactuals FOXES ARE MORE WILLING THAN HEDGEHOGS TO ENTERTAIN SELF-SUBVERSIVE SCENARIOS
    (pp. 144-163)

    There is something disturbing about the notion that history might turn out to be, as radical skeptics have indefatigably insisted, one damned thing after another. And there is something reassuring about the notion that people can, if they look hard enough, discover patterns in the procession of historical events and these patterns can become part of humanity’s shared endowment of knowledge. We need not repeat the same dreadful mistakes ad nauseam and ad infinitum. Here is a bedrock issue on which hedgehogs and foxes can agree: good judgment presupposes some capacity to learn from history.

    The agreement does not last...

  10. CHAPTER 6 The Hedgehogs Strike Back
    (pp. 164-188)

    It requires better defense counsel than the author to get the hedgehogs acquitted of all the charges against them. Too many lines of evidence converge: hedgehogs are poor forecasters who refuse to acknowledge mistakes, dismiss dissonant evidence, and warm to the possibility that things could have worked out differently only if doing so rescues favorite theories of history from falsification.

    That said, any self-respecting contrarian should wonder what can be said on behalf of the beleaguered hedgehogs. Fifty years of research on cognitive styles suggests an affirmative answer: it does sometimes help to be a hedgehog.¹ Distinctive hedgehog strengths include...

  11. CHAPTER 7 Are We Open-minded Enough to Acknowledge the Limits of Open-mindedness?
    (pp. 189-215)

    Chapter 6 revealed considerable canniness in what looked like incorrigible closed-mindedness. It failed, however, to exonerate experts of all the cognitive indictments against them. In this chapter, therefore, let us assume a lingering problem: all too often experts, especially the hedgehogs among them, claim to know more about the future than they actually know (chapter 3), balk at changing their minds in the face of unexpected evidence (chapter 4), and dogmatically defend their deterministic explanations of the past (chapter 5).

    The diagnosis implies a cure: observers would stack up better against our benchmarks of good judgment if only they were...

  12. CHAPTER 8 Exploring the Limits on Objectivity and Accountability
    (pp. 216-238)

    Objectivity was the bedrock principle on which professional societies of historians and social scientists were founded in the nineteenth century. The disciplinary mandate was to move closer, by successive imperfect approximations, toward the truth, a truth unadorned by mocking quotation marks. Well before the twentieth century’s end, however, scholars started chiseling at this foundation, raising doubts about the positivist project and the feasibility of sharp distinctions between observer and observed, fact and value, and even fact and fiction.¹ Constructivist and relativist epistemologies—which depicted “truth” as perspectival and demanded to know “whose truth”—garnered considerable respectability.

    My own sympathies should...

  13. Methodological Appendix
    (pp. 239-272)
  14. Technical Appendix
    (pp. 273-312)
    Phillip Rescober and Philip E. Tetlock
  15. Index
    (pp. 313-321)