Survey Limitations

 

I'm being driven crazy by survey requests. They are turning up everywhere. I get several a day via email. Every time I buy something, the receipt includes a link to do their survey. On-line purchases ask me to do a review. I've now established a policy of "I don't do surveys", end of discussion.
As a former student of psychology the whole process has annoyed me because of the way marketers seem to blindly accept the information as 100% reliable. It has always appeared to me that marketers work to such a low standard as compared to how scientists use surveys and then set up experiments to test what people tell them.
In the famous words of House, "people lie". Recently, a friend told me when a survey gives you an incentive for completion (a discount or a contest entry) she does the survey and purposely makes selections which are untrue. I got such a good laugh out of this as she took such delight in telling me how she was "sticking it to them".
Paco Underhill who is an environmental psychologist and the author of many books on shopping is a fascinating read if you are interested in this topic. His theories were developed and based on hard data collected by a team of researchers using thousands of hours of field research; in shopping malls, department stores, and supermarkets in the U.S.
So why do companies keep asking for surveys? Because it's cheap compared to the scientific methods. It's also fast but is it reliable?

Probably not, if the example of my friend indicates. I'm sure you've noticed surveys often include leading questions when you have a multiple choice that doesn't include the answer you would like to give. It's also a well known phenomenon in the science world that surveys are influenced by the sample and the people who will do your survey may be very different from those who won't. Marketers need to acknowledge if their target audience won't participate, they'll never get the data they want.
There's also the Hawthorne effect, in that research subjects know they're being observed, and this awareness can alter their behaviour. Another friend tells me he always chooses the highest income level bracket on any survey because "it's none of their business!"

Despite their attempts to be neutral, marketers will always provide subtle verbal and nonverbal cues to subjects. This can influence a subject's thoughts and behaviors. All good researchers are conscious of how difficult it is to avoid influencing test subjects. I doubt marketers get the extensive training to avoid this. Marketers also have an agenda, collecting information is the goal for payment, accuracy may not be as important. This is why paid research in the academic world is always heavily scrutinized. Have you every asked a question about a survey and been told "just answer as best as you can"? Your question could be critical to their understanding yet the data collectors aren't interested.


So what about you, have you done any surveys lately?