Planning to Err?
Then do it as publicly as possible
Franklin D. Roosevelt
|Literary Digest calls the 1936 election for Alf Landon (remember him), in an electoral college landslide. The poll was performed by sending postcards to people with telephones, magazine subscribers, car owners, and a few people on lists of registered voters. The first problem was that the sample was biased toward Republican-leaning voters, who could afford magazine subscriptions and telephones during the Great Depression. The second problem is the response rate (defined) error: they receive only 2.3 million responses, for a 23% response rate. This was compounded by a volunteer error (defined): the respondents are most likely people who wanted change, and Roosevelt is president. Nevertheless, Franklin Roosevelt, the Democrat, wins in a landslide.
Statistician Jessica Utts (see Seeing Through Statistics, pages 65-66), who examined issues of the Literary Digest from 1936, says "They were very cocky about George Gallup predicting they would get it wrong. [Gallup helped make his reputation in polling by correctly calling the race.] The beauty of something like that is that the winner is eventually known."
- Aided by erroneous polling, newspapers prematurely call the presidential election for challenger Thomas Dewey, leading to the famous photograph of an elatedly re-elected Harry Truman. The problematic polls use then-popular "quota-sampling" techniques. In other words,
Harry Truman AP/Wide World Photos
- they sought out a certain number of men, a certain number of women, and similarly for blacks, whites, and various income levels. According to statistician Fritz Scheuren, quota sampling in political polling was abandoned after this debacle in favor of random sampling of the population.
- Polls in the United Kingdom misgauged the Conservative Party's victory by between 7 and 11 percent (see Polls Apart). But things may be getting even worse: some polls published in August, 1995 reported the Conservatives were trailing Labor by 13 percent; others reported 35 percent. What accounted for the difference? It was adjustments used by one polling firm to try to make the poll numbers reflect the fact that many people don't like to report unpopular opinions.
- A 1992 poll commissioned by the American Jewish Committee finds that 22 percent of the public thought the 1939-1945 Holocaust of world Jews in Europe never happened. The poll exploded like a bombshell for those who had lived through the horror. But a subsequent poll found that between 2 and 9 percent of Americans thought the Holocaust had not happened.
What happened? It was a simple case of convoluted wording, chosen to reduce bias, said Burns Roper, head of the Roper polling organization, which carried out the poll. The question: "Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened."
"We should have never approved the question, and we certainly never should have written it," said Roper. In early 1994, The Gallup Organization tested three wordings for the question and found that
- 33 percent thought it was "possible" that the Holocaust had not happened, using the Roper wording.
- 9 percent doubted the existence of the Holocaust, using better wording.
- But with clearer wording, fewer than 0.5 percent said the Holocaust "definitely" did not happen, and 2 percent said it "probably" did not happen.
- ABC, CBS and CNN are forced to admit that they relied on incorrect exit polls when they predicted that Sen. Bob Dole would finish third in the Arizona primary (Dole finished second). The mistake was widespread because the networks relied on data from the Voters News Service. Pollsters said the error crept in because followers of Pat Buchanan actively sought out pollsters, and that exaggerated Buchanan's actual vote. Tom Hannon, CNN's political director, told the New York Times (see 3 Networks Admit Error...) that voter polls in primaries have shown a 3 percent bias in favor of Buchanan -- enough to account for the recent mistake. That's another case of the volunteer error.