Market Research: Can you trust it?
Hal Varian, chief economist at Google recently said that the “the sexy job in the next 10 years will be statisticians”.
I think he’s right. Every participant in a first-world economy should have a solid understanding of stats. Not that I’m recommending that everyone become a statistician, but with the volume of information out there, it’s important to understand how data can be used to sway you.
Here are two “statistics” about my own life:
i) I was the only female graduate in Pure Mathematics at my university in my year (true).
ii) 50% of my graduating class who majored in Pure Mathematics were women (also true).
One of those statements paints a picture of a mathematics education that is oppressive to women, somehow subtly driving females away. The other paints a picture of a very progressive math department that gave women the same opportunities as men.
As I’m sure you’ve figured out, there were only two people in my graduating class that majored in Pure Mathematics. But without that information, I could have easily mislead you. Clearly the statistical significance is not there.
When someone presents information, whether a market researcher or anyone else, always think about the reliability. What’s the methodology? How big is the data set? Is the audience who completed the research the right audience to comment on the topic? Is it representative?
Research bias is very real. You need to be aware of this and always ask the right questions in order to determine whether information presented to you is valid – whether selling enterprise software or listening to the media.
3 comments
This is a great point. A lot of times (I’m guilty as well) when we read survey findings/reports we forget to ask how you got there (methodology), and what information is missing (what did you ask that you’re not sharing — or what didn’t you ask that you should have). This principle certainly stretches beyond just market research to other types of survey research.
It is incredible to me how few managers and professionals have a basic grasp of statistics. In my line of work (load testing), most people make decisions based on small differences between runs because they do not understand the concept that random variation could be responsible for the difference. There is a self-feeding vicious cycle where people will continue to make deceptive statements until they are challenged by properly educated consumers.
It’s so easy to be swayed by snazzy marketing when you aren’t comfortable with numbers. The fear keeps you from asking important questions. It’s a tough job for researchers and non alike.
Leave a Comment