Can Polls Be Trusted?

Can polls be trusted? My post yesterday on the Pew poll on Libya drew several excellent responses that raised just this question. The tradition is to reply to comments in the comments box that follows each post. But these responses raised some broader issues that merit a sustained response.

Allyson and Florencio De la Cruz both raise the issue of question wording: How a question is asked can influence the responses you get. To quote Allyson:

I would really have liked to see a question directly related to support for a NFZ [no-fly zone] if used to protect civilians. Intervening with the goal of supporting one side of an armed conflict is much different than taking action to prevent innocent men, women and children from being killed by the Libyan air force.

Florencio argues that Pew posed a

Biased poll question—the word “responsibility” is the culprit. Most do not think that the U.S, has a responsibility, but that does not mean that the U.S. shouldn’t for other reasons, like say, long-term strategic interests in the region. The Pew poll is silly.

Yes, question wording is a problem that bedevils pollsters. Take for example the recent CNN/Opinion Research Corporation Poll, which as Allyson noted in her comment, found greater support for a no-fly zone (56 percent in favor to 40 percent opposed) than did the Pew poll (44 percent to 45 percent). Differences in sampling techniques (that is, who got asked) and question ordering (that is, where a question appears in a survey and what other questions it follows) may help explain the difference. (The CNN and Pew polls were done at the same time so we can rule out the possibility that the respondents were responding to different events in the news.)

But question wording almost certainly played a role. The Pew poll question is brief:

Would you favor or oppose the United States and its allies doing each of the following with respect to Libya….enforcing a no-fly zone over Libya?

The CNN question is longer and provides more background information:

Some people have suggested establishing a “no-fly zone” in Libya which would be an area patrolled by military planes from the U.S. and other countries to prevent Gadhafi from using his air force. No U.S. ground troops would be involved but U.S. airplanes or missiles might be used to shoot down Libyan airplanes or attack ground bases used by the Libyan air force. Would you favor or oppose the U.S. and other countries attempting to establish a no-fly zone in Libya.

This formulation mentions Qaddafi by name, assures the respondent that the no-fly zone won’t expand into a ground war, and notes that Libyan (but not U.S.) planes could be shot down. All this information tilts responses in favor of the no-fly zone. If you doubt that, then do this thought experiment: Would support for the no-fly zone go up or down if the lead up to the question said that the United States would be intervening in a civil war, U.S. planes might be shot down and American pilots killed, and that Qaddafi probably would still defeat the rebels.

I have been involved in many survey efforts, so I know first-hand that pollsters—at least good ones—work hard to write questions that are both precise—so that the responses tell us something we want to know—and neutral—so the questions don’t steer respondents to an answer. That is far easier said than done.

To see how hard it is to write good questions, take the question that Allyson poses. Any question that asks whether people want to act to save innocent lives practically begs for a “yes” response. At the same time, Allyson’s preferred question presumes that intervening to save innocent lives is different than intervening on behalf of one side of an armed conflict. But intervening to save lives is to choose one side of the conflict.

Florencio’s response is to dismiss the Pew question as “silly.” But this goes too far. The Pew question is legitimate, even if only because opinion leaders will frame question in just this way when making their case to the public. Put differently, if Florencio is right that mentioning “responsibility” makes people rethink intervention in Libya, then that’s a word that people who oppose intervention will use to tilt their audiences in their favor.

One way to avoid being misled by question wording is to do what Florencio did and look for words that might bias responses one way or another and then discount the question appropriately. Another way to avoid being misled is see what different polls have to say on the same topic. Different pollsters inevitably end up wording questions slightly differently, especially when an issue has just emerged in the news and there hasn’t been time for a consensus to emerge on what constitutes the ideal question. But taken together, different questions provide a good sense of where the public stands on an issue.

Here a comparison between the Pew and CNN polls is instructive. Pew found that 82 percent opposed sending in ground troops, while CNN found 76 percent opposed. Pew found that 63 percent of those surveyed don’t believe the United States has a responsibility to do something about the fighting in Libya. CNN asked a different question that got at the same point:

Do you think the United States should take a leading role in any attempts to resolve the situation in Libya, or do you think the U.S. should leave it to international organizations or allies such as Great Britain or France to take the leading role in Libya?

CNN found that 74 percent opted for the “leave it to others” response.

These two polls, along with the Rasmussen poll all point to the same conclusion: Americans right now are wary of direct military intervention in Libya. Externality’s response to yesterday’s post provides one set of reasons why that is the case.

Of course, polls are no more than snapshots in time. Public opinion can change and change dramatically in response to events and presidential leadership, as a TWE Remembers post that I hope to get up tomorrow will show. But for now I will leave you with a question: Do you think Americans will be more or less inclined to support a U.S. intervention in the Libyan crisis should Qaddafi’s forces reach Benghazi and cable news, the internet, and twitter carry the fighting live?