The billion is the new million. A million used to be a lot. Nineteenth-century Americans borrowed the French termmillionnaireto denote those whose wealth had reached the astonishing total of a million dollars. In 1850, there were 23 million Americans; in the 1880 census, New York (in those days that meant Manhattan; Brooklyn was a separate entity) became the first U.S. city with more than one million residents.

At the beginning of the twenty-first century, a million no longer seems all that big. There are now millions of millionaires (according to one recent estimate, about 8.6 million U.S. households...

Having a small store of factual knowledge prepares us to think critically about statistics. Just a little bit of knowledge—a few basic numbers and one important rule of thumb—offers a framework, enough basic information to let us begin to spot questionable figures.

When interpreting social statistics, it helps to have a rough sense of scale. Just a few benchmark numbers can give us a mental context for assessing other figures we encounter. For example, when thinking about American society, it helps to know that:

The U.S. population is something over 300 million (about 312 million in 2011).

Each...

Some bad statistics are the products of flubs—elementary errors. While some of these mistakes might involve intentional efforts to deceive, they often reflect nothing more devious than innocent errors and confusion on the part of those presenting the numbers. For instance, after Alberta’s health minister told students at a high school that they lived in the “suicide capital of Canada,” a ministry spokesperson had to retract the claim and explain that the minister had spoken to a doctor and “misinterpreted what they talked about.” In fact, a health officer assured the press, the local “suicide rate is among the...

While it is sometimes possible to spot obvious blunders, most statistics seem plausible—at least they aren’t obviouslywrong. But are theyright? In trying to evaluate any number, it helps to ask questions. A good first question is, Who produced this figure? That is, who did the counting—and why?

Numbers don’t exist in nature. Every number is a product of human effort. Someone had to go to the trouble of counting. So we can begin by trying to identify the sources for our numbers, to ask who they are, and why they bothered to count whatever they counted....

Every statistic is the product of somebody’s counting something. The previous section shows that we need to ask who did the counting, and why they bothered to count. This section turns to another very important issue: what did they count, and how did they decide what counted?

Counting requires dividing the world into those things that will be counted and those that won’t. Suppose we want to count the number of children in poverty. We need to begin by defining what does and doesn’t count as a child in poverty. What is a child? Is it someone under 16? Under...

After defining what they want to count, the people who wish to produce statistics must actually do the counting. To do so, they must devise procedures or methods that will allow them to measure whatever they wish to count. The choices they make raise another fundamental question: How did they go about counting—that is, measuring?

Suppose someone wants to measure public attitudes toward gay marriage. An obvious way to do this is to survey the population, but immediately a number of practical issues arise, including how to select the people to respond to the survey, how to conduct the...

The sources for our statistics find themselves competing for the attention of the press, policymakers, and the general public. Savvy advocates work to devise claims interesting enough to make the cut. They may point the press toward disturbing examples, give journalists interesting demonstrations to cover, get a celebrity to endorse their cause, and—of course—present arresting numbers.

Similarly, the media are looking for ways to turn the stories they do decide to cover into compelling news. Once again, statistics can play an important role because they convey the sense that a story is grounded in hard facts. But the...

Some statistics are products of bureaucratic routines. For instance, each month, the Bureau of Labor Statistics releases a report on the level of unemployment, just as the Census Bureau gives new population figures after a new census is taken every ten years. These are the social scientific equivalents of the National Weather Service’s daily reports on temperature and precipitation. The figures provide pretty good information—based upon well-understood definitions and measurements—about shifts in the economy or the population or the weather. They are, if you will, just information.

But people often use statistics with a purpose in mind, as...

Many of the statistical examples we’ve been considering so far stand more or less alone. A news report about a social problem often contains a single statistic—a billion birds a year die from colliding with windows, for example. In such cases, the work of assessing the number falls to us. We can accept the figure, ignore it, or think critically to determine how much confidence we ought to have in it.

However, on other occasions we may encounter competing statistics: different sources—opponents in a debate—may present very different numbers, often also criticizing the opposition’s figures. Each side...

This field guide has classified a number of common flaws in the sorts of statistics we encounter in media coverage. These are things we’ve learned to watch for:

Numbers seeminconsistent with benchmark figures(basic, familiar facts) (b.1)

Severe examplesare used to illustrate a supposedly common problem (b.2)

Numbers that seem too high or too low may be caused by amisplaced decimal point(c.1)

Botched translationsconvert statistics into simpler but incorrect language (c.2)

Misleading graphsdistort the reader’s visual impression of data (c.3)

Errors in strings of calculationsaffect the final figures (c.4)

Big round numbersmay...

Reading a field guide like this might lead you to conclude that all numbers are bad numbers—that you ought to approach every statistic with a cynical attitude and simply assume that figures are worthless. But that won’t work. Our world is complicated. We can’t hope to understand it without trying to measure its properties. We need statistics—but we need good ones, as accurate as possible.

All statistics are the products of people’s choices. If they’d made different choices, the figures would be different. This is inevitable; we can’t get around it. But with enough information, we should be...

Every statistic is the product of a series of choices made by the people who produce, process, and report the data. In particular, when we see statistics in media coverage, we need to appreciate that those numbers are the products of choices made not just by the folks who actually gathered the data but also by those who brought the story to the attention of the media and by the people in the media who selected this story for coverage and who then chose how to repackage the information as news. Different statistics are the results of different choices, made...

By now it should be clear that there is no shortage of questionable statistics: any newspaper, magazine, or news broadcast is likely to present at least one number that may give you pause. Happily, there are lots of people engaged in evaluating statistics and thinking critically about the role numbers play in our society. Here are just a few sources you might enjoy examining.

FiveThirtyEight: Nate Silver blogs about political polls (and assorted other data-related topics); he is especially good at explaining the choices that lead to different polling results. Managing to be both readable and sophisticated, this is the...