Tuesday, August 7, 2007

Survey of Surveys

Mark Twain Presents: A Survey on Surveys


“There are lies, damn lies, and statistics”

Mark Twain’s famous quote is still as valid today as it was a century and a half ago. Probably even more so as most American citizens are inclined to trust and accept blindly the results of “surveys” no matter what they may be or how they were conducted. Having performed many in my career and mentored more performed by students, perhaps a basic course in surveys and their potential misdeeds is in order.

1. The results are only as good as the data and processes used. Computer Freaks have an acronym that describes this perfectly: GIGO: Garbage In, Garbage Out. Unless you are meticulous and conduct the research through systematic endeavor, the results will indeed be GIGO. For example, a Manhattan liberal conducts a survey of twenty of his friends. He determines one out of twenty voted for Bush, perhaps two attend church occasionally or regularly, and all twenty agree that America is the real enemy, not Iran. If he were write up the report and issue it as a survey result of the entire American population, it would be numerically correct but totally meaningless. In this case, the sample chosen was a convenience sample and was not appropriate for the population (the population here being the American citizenry). In any valid survey, the sample surveyed must approximate the population as a whole. This does not take millions of people. Unemployment labor statistics for entire states are valid based upon the sampling of a few thousand. The sample must approximate the demographics of the group and be random in nature to be appropriate.

2. You can basically determine the outcome of any survey very simply by manipulating the wording of the survey questions, that is On how the questions are phrased. The proverbial “Have you stopped beating your wife?” portrays it best. Wording matters. If a pollster asks, “Are you concerned that escalating and increasing surface temperatures brought on by man=made gasses are melting the icecaps at an ever increasing rate and could raise the sea level by hundreds of feet?” should we be surprised at a 95% Agreement. Of course the survey result is then echoed as “95% of Americans worried about global warming.” Which was not the question asked. Adding in strong terms, biased phrases, loaded language (as in the wife beating question above) will result in manipulating the result to what is desired. This might be good for your cause but it is lousy science and certainly not appropriate research.

3. The Encyclopedic survey: whereas everyone wants to add his own particular question and soon the survey becomes the size of a SAT test booklet. The more questions added, the less likelihood respondents will wish to take the test or respond honestly (and the higher the chances they will soon begin marking in the answers without regard to the questions or options much as a test taker does when they have two minutes and fifty questions left). I have seen a questionnaire with nearly a thousand questions that would take hours to respond properly to. I wonder to myself who has the time to commit to the survey, let alone answer it honestly. And those that do, are they representative of the population you wish to review? Surveys must be kept short, simple, understandable, and easily completed in five minutes or less. Otherwise, the short attention spans of today’s adults will begin to wonder and any results you get will probably be worthless.

4. Be careful of statistics. A probability of 5% means the likelihood of occurring by chance is 5 percent. If you have twenty such questions and use that measure, you have a good chance of finding something “significant” that is there simply because of chance.

In conclusion, the next time you see the results of a survey and they seem counter-intuitive to you or as if they had surveyed inhabitants of Mars, not Earth, you must ask yourself a) was the sample representative of the population, 2) what was the actual survey like and how were questions worded and ordered, and 3) were the statistics meaningful? And likewise when you as a businessperson is conducting any research or survey, be careful, since, if not done properly, it is a total waste of time and money and could very well guide you in the wrong direction than is actually the case.

No comments: