Friday, August 20, 2010

Lies, Damn Lies and Surveys

This past week we have been inundated with polls showing that 20%-25% of Americans think President Obama is a Muslim, a quarter believe he was born in another country, and 45% think "building a mosque at ground zero" is an "insult."

As I was mentioning this morning in class regarding survey research, the first thing you should pay attention to when being surveyed is the wording of the question. Is the question valid? Is the researcher leading the respondent to a fallacious (or in this case, sensationalist) response?

A closer look at the Pew, CNN, and Time magazine surveys shows, indeed, that the questions asked regarding the president's religious beliefs and birth background were flawed.

As the WaPo article notes, measuring "misperception" can be tricky because you are trying to quantify a negative. Still, the Pew organization should really know better. Here was their question:

"Now, thinking about Barack Obama's religious beliefs ... Do you happen to know what Barack Obama's religion is? Is he Christian, Jewish, Muslim, Buddhist, Hindu, atheist, agnostic, or something else?"
Gack. I wouldn't allow underclassmen to ask such a misleading question, much less graduate students. The premise of the question assumes the respondent doesn't know "Barack Obama's relgious beliefs," thus creating suspicion and doubt. The logical response to that question would be, "well gosh, I thought I did, but maybe I don't...hmmm." As the WaPo piece points out, following this with a closed-ended set of choices limits the respondent's line of thinking and leads to possible false answers.

As an aside: I wonder how many respondents would have chosen "who gives a flying f*$#" if offered the choice?

The Time Magazine questions are even worse:
"Do you personally believe that Barack Obama is a Muslim or a Christian?"
Don't you love the dichotomous choice, and the use of the qualifier "personally"? And by frontloading Muslim before Christian, you guarantee a disproportionate first choice response (such as people who vote for the first candidate on a ballot when they don't know anyone who's running).

Check out the mosque question:
"Some people say that building the Muslim community center and mosque near the World Trade Center site would serve as a symbol of the country's religious tolerance. Others say that building the mosque near the World Trade Center site would be an insult to those who died on 9/11. Which comes closest to your view, or do you agree somewhat with both views?"
The ubiquitous "some people" qualifier makes the question invalid right from the start. By framing the question in an either/or format, you force the respondent to choose one of two extreme answers, and given the inflammatory choice "insult to those who died on 9/11" coming last, you guarantee a loaded response.

And on whether the president is a "foreigner", there's this gem from CNN:
"Do you think Barack Obama was definitely born in the United States?"
LOL. My 8 year old asks more cogent (and less leading) questions. The premise of the question creates the doubt. As with the above, it suggests to the respondent "well, I thought he was, but maybe he wasn't?"

So what is the point of all this (other than to suggest that the people writing/asking these questions are as dumb as the respondents being portrayed)? That we should be careful about judging the veracity of public opinion being represented in the mainstream media.

This isn't to suggest that healthy numbers of Americans don't believe these things (better worded questions might tell us). But it is to suggest this kind of opinion polling furthers the fallacy of personal attack.

Debating the administration on the issues is one thing; debating the president's personal virtues (and whether he's "secretly one of them") detracts from the real issues.

Which, of course, really mean nothing in an election-year.

No comments: