(Thanks to Cathy Harrison and  Bernie Malinoff for Tweeting this article.)

Medical and psychological researchers have strict ethical guidelines in place dealing with people in their research, maybe people who run Web surveys need some also.  Here is our input into guidelines that should be included in a manifesto for being kind to human participants in market research studies.

1. Never force someone to give you a wrong answer – When Steve Carrell gave Conan O’Brien his exit interview on the Tonight Show and asked if working at NBC was “great”, “really great” or “fantastic”, I had a flashback to real-life employee survey I had to take years ago.  Web surveys should always cover every possible option someone could give or offer an “other” option.  A good survey will have a comments area at the end to allow users to clarify anything they felt didn’t represent their input.
2. Use complex ranking or rating matrices with caution – It is just plain painful to answer long rating or ranking questions even when you deeply care about the subject. You always end up missing at least one line and get that oh-so-not-helpful “please answer number 14″.  Be kind and use matrices sparingly and only when there are no other approaches to get the answer you’re looking for.
3. Don’t ask questions that simply don’t matter – Seriously, does my ISP need to know if I’m a man or a woman?  Is that going to change the way that they deliver service to me?  If it does, then by all means ask, but I frequently feel like somebody took a market research 101 class back in their first year of college and ever since have opened with demographic questions on age, gender, and ethnicity independent of if it matters to the survey goals.  There is something to be said for asking “warm-up” questions that are easy to answer, but surely there is something more interesting then gender to ask.  [UPDATE:  Of course there is always the "Elinor Exception" to this rule as rightly pointed out in the comments.  If you need to add a sanity check on the quality of the sample, then go ahead and ask the gender question.]
4. Don’t expect participants to know all your lingo – Seriously.  Speak plain English  (or French or Spanish or Chinese or Japanese or whatever language your participants do).  Don’t throw in tons of jargon without defining it first.
5. Be realistic about how long your research takes – It seems as if every survey in the world is “10 minutes” long.  Surely you can have at least one person who didn’t write the survey questions try it out and tell you how long it realistically takes an average person.

I’m sure at Dimensional Research we have on occasion broken one of these rules, but we try our hardest to ensure our participants don’t suffer in the process of giving their feedback.  We’re always happiest if they actually enjoy it.

What would you add to a “Manifesto on Ethical Treatment of Survey Takers”?

A very simple example of a subtle guidance: When taking personality tests, if you say “take your time” you get different responses than if you say “do your best”. “Take your time” implies that you want very thoughtful answers, and people respond accordingly. “Do your best” is neutral and doesn’t have  implicit guidance built in.

Think about your research goals, and set up your questions appropriately.

If you’re doing a message test where in real life people don’t take a lot of time to think through underlying ideas, create a similar environment with guidance like “Please review and give me your first gut reaction” or “Don’t over-think this, I’m looking for your initial response.”

If you’re looking to understand pain, it may take some digging to get to the source because your participant may not actually be aware of what’s going on.  This is particularly true with technology professionals that have lived with the pain for a while and have a workaround in place. They may not notice how much time they spend doing something that is not value added because it’s just the way it works.  In that case, ask questions that encourage  thoughtfulness, such as “Why is that?” or “Really, tell me more about that” or “Could a different approach be more valuable?“

I think he’s right.  Every participant in a first-world economy should have a solid understanding of stats.  Not that I’m recommending that everyone become a statistician, but with the volume of information out there, it’s important to understand how data can be used to sway you.

Here are two “statistics” about my own life:

i) I was the only female graduate in Pure Mathematics at my university in my year  (true).

ii) 50% of my graduating class who majored in Pure Mathematics were women (also true).

One of those statements paints a picture of a mathematics education that is oppressive to women, somehow subtly driving females away.  The other paints a picture of a very progressive math department that gave women the same opportunities as men.

As I’m sure you’ve figured out, there were only two people in my graduating class that majored in Pure Mathematics.  But without that information, I could have easily mislead you. Clearly the statistical significance is not there.

When someone presents information, whether a market researcher or anyone else, always think about the reliability. What’s the methodology?  How big is the data set? Is the audience who completed the research the right audience to comment on the topic?  Is it representative?

Research bias is very real. You need to be aware of this and always ask the right questions in order to determine whether information presented to you is valid – whether selling enterprise software or listening to the media.

Seriously.

While it’s very human to only want to hear how wonderful your solution is, you need to learn about any objections to your product NOW. Not later, when you have to deal with an entire slipped pipeline, because you didn’t realize there was a deal-breaker.

You need honest feedback, and you need it now.

But how do you ensure feedback that is real? Very few people, no matter how stupid they think your idea is, will actually tell you that. You have to create an environment that allows people to tell you about the downside. 

One of the more painful exercises you can go through is to put a group of people together in a focus group and ask “what DON’T you like about this solution?” Once you ask them this question, and give them permission to be brutally honest, they WILL tell you about everything they don’t like. And the list of things they don’t like will be very long. And their list will inspire the other participants to think about even more things that they don’t like.

But as I said, this is GOOD FOR YOU!  If you know about the objections, you can prepare for them in advance. In the long run, this will save you significant amounts of time and money.

I’ve discussed here the importance of asking market research participants open-ended questions such as “can you tell me more about that?” and “is there anything else?”  I also written about the importance of getting honest feedback, and suggested ways to make sure you do.

Today I’d like to emphasize the importance of giving the participants permission to NOT know.

Market research participants are smart, competent people who know what is going on in their environments. It’s natural that they’re only human and don’t want to come across as unknowledgeable. Nobody wants to admit, “I don’t know.” 

However, YOU need to know the truth. If they don’t know about a product, you need to know about that and adjust your marketing efforts accordingly. It’s just as important for you to uncover that participants are unfamiliar with a product or an idea as it is to know their opinion about it.

The key is to ask questions that allow a participant to tell you the truth without feeling like an idiot.

A question that is in many research guides is an awareness question. “Are you familiar with product X?  What do you know about it?”  I don’t like asking the question this way. It almost guarantees that some selection of the participants will answer, “Yes, I’ve heard of it. I read about it in a magazine” just so they don’t have to admit they have never heard about the product.

Instead, ask “Are you familiar with product X – and please feel free to say no, that’s just as important for me to know.” This will give the participant permission to not know without feeling stupid. 

Another option is to ask, “What do you know about the following product – if anything?” that also leaves the door open for participants to admit they don’t know without worrying about disappointing you.

According to a recent report, “Did not listen to me” is the most widely experienced problem faced by 38% of services buyers. Additionally, 55% of buyers said they would be “much more likely” to consider hiring the provider “if they listened better.”  As Drew McLellan points out, if you add up the “didn’t listen” and the “talked too much,” that’s 63%.  Two thirds of your sales calls are being wasted simply because you talk too much and don’t listen enough! 

Market research is all about listening. It is also about asking the right kind of questions so that you’ll get the right kind of feedback. 

Here are three tried and true tips that I always use in my market research to make sure I actually listen to participants. These tips are applicable to every area of customer and prospect interaction.

1. Shoot for the 80/20 rule.  Always ask yourself, what percentage of the time are you talking, and what percentage of the time are the participants talking?  I shoot for the 80/20 rule.  If I’m doing more than 20% of the talking, then I’m probably not really listening.
2. Ask questions that encourage talking – not just “yes/no”.  Are you telling the person what their pain is, or are you letting them tell you about their pain?  A good researcher doesn’t say, “here is your pain – isn’t that right?”  They ask about pain to see if the pain they believe the prospect has actually comes up as a priority.  Of course you can always guide a discussion to keep it relevant, but you need to ask open questions in order to get real feedback – and give customers and prospects a real chance to say “no, I don’t care about that”.
3. Look for what you don’t know.  Never ask a participant “Do you agree?” 90% of the time they’ll say yes, because that’s the easy and polite thing to do. Your goal is to create an opportunity to hear something you DON’T know. 

To quote the Greek philosopher Epictetus, “we were born with two ears and one mouth, so we can listen twice as much as we speak.”  Thanks for the inspiration Drew.

You are very, very fortunate if you have customers that will actually give real, honest feedback in a scenario like that. They want to keep things positive and upbeat – they just saw pictures of that product manager’s kids or heard about their latest adventure trip. They don’t want to get all negative, even if that is what they really think.

One of the things that market research does really well is to intentionally CREATE an environment for honest feedback and for uncovering information you haven’t anticipated.  To create such an environment, you need to:

            1) Make sure the conversation is not about the personal relationship. Of course you want the relationship, and you want to leverage that. But the feedback can’t be influenced by whether someone likes you or not.  Of course they like you – you’re a tremendously likable person!  But the feedback must be about the topic, the market, and the pain points, not about what they think about you. 

You need to use verbal and non-verbal cues to set up the discussion so that it is open, non-judgmental, and the client is never aware that what they’re saying is bad news for you. You need to get the bad news now – not after you’ve been through a long, involved sales cycle that ends because there is “no budget,” which really means “not enough pain.”

            2) Dig for negative feedback.  Don’t just ask, “what do you think?”  Encourage people to talk about what they DON’T like.  You can’t make it better if you don’t know what’s wrong.  Ask, “What do you LIKE about what I just showed you?” then ask “What DON’T you like about what I showed you?” 

            3) Put feedback in context.  It’s not enough for a user to connect with the pain you solve – it has to be a priority!  I recently was in a focus group where a participant described a situation troubleshooting a particularly elusive problem in a complex application. It was a dramatic story with dozens of IT staff from different technology groups getting on bridge calls, pointing fingers, escalating issues, and finally getting the CIO involved in a 48-hour saga that ended up to be a rather minor configuration issue. I could feel the client behind the glass get excited because this was exactly the type of problem they envisioned their solution solving.

But then we asked how important it was to solve this kind of issue so it didn’t happen again. The participant explained that it wasn’t important at all.  It was the first time in 15 years it had happened and they had solved it.  They had moved on.  The pain wasn’t a priority.  Fortunately that group did express other pains that the client could solve, and they learned important lessons on ways of positioning their product that matched the most important pains.

Employing these techniques a great way to encourage your customers to give you real, honest feedback and to help you uncover pains you were unaware of.

Treat “unprompted” feedback with special importance.

Sometimes the information you don’t know that you need to ask about, is the most important information to hear.  Of course research must have a goal – but work with your researcher so they know your business and can use their judgment to understand what threads are worth digging into and what will take them into a rat hole that does not forward your business. 

The most important tool for unprompted feedback is this question:  “Is there anything else?”  This is the final question Dimensional Research uses for any market research interaction. After you’ve asked all your carefully crafted and specific questions, closing with an open-ended question can yield incredibly useful information.

Of course, this is a very important balancing act and research projects need to focus on their goals – not interesting side topics.  The core questions must be asked. But Dimensional Research makes a point of going through all input from every project with a special eye for the unprompted information.  Most of it is one-offs, with each participant having a special thing that only they are interested in.  But often you can spot trends in unprompted information that are really important. 

We’ve done projects where it turns out there was a critical development tool that needed to be integrated with the client’s product – not something they had realized was vital until the majority of participants talked about it, unprompted, in a series of customer and prospect focus groups.

In another project we didn’t realize that customer loyalty was almost non-existent, until we went through the unprompted feedback from interviews with the highly satisfied, happiest customers, and discovered they regularly evaluated the competition – not a question that we asked about specifically.

So do stay on topic.  But keep your ears open for the “+1″ opportunities or customer issues you hadn’t recognized – especially if you hear them from multiple sources.  Possibly minor product changes or messaging tweaks can help you address a different pain, target a new and lucrative market, or address important weaknesses.

When it comes to uncovering actionable information, often the best information comes from answers that the client has not predicted. While there is tremendous value in validating a known theory, it’s very useful to be surprised with a new perspective.  My absolute favorite way to uncover these unpredictable nuggets are these two questions:

• “Can you tell me more about that.”  This is a fantastic follow up question.  It does not lead a participant in any way, it just opens up a way to hear more. Even if the top-level answer is exactly what you expect, using this as a follow-on can uncover motivations that you never expected.
• “Is there anything else?” This is the last question in any guide that I write.  After you have spent time talking to people, there is often something really pressing that they’re just waiting for you to ask and you haven’t.  This is a great question for gathering “top of mind” issues.  And of course, always try the one-two punch of “Is there anything else?” combined with “Tell me more about that” for real hidden gems.

On the flip side, there are just some questions that don’t uncover anything actionable.  People are great at describing their challenges and giving you feedback on proposed solutions, since they know their processes and organizations.  But there are things people do that they aren’t really aware of.  Two areas where I find direct questions don’t work very well are:

• “How much would you pay for this?”  Every time I have heard that question asked there is always at least one person who says, “Frankly, it should be free” and another person who says “Millions!”  People do not do a good job of assigning value to products on their own.  A much better approach is to try and get historical examples of pricing.  Find some “anchor reference” product – a competitor or a complementary product – and determine the price they paid and how that purchase cycle worked. Once that conversation has produced a reference price, have the participants connect the dots between the known and unknown product.
• “How should we market this product to people like you?”   Unless you’re selling to marketing people (which our clients almost never are – we are technology experts) this question never results in anything actionable.  People don’t think about how they absorb information.  The most common answer to this is “I Google if I have a question”, but when pressed they can’t give examples of keywords or phrases they’ve used, or what they clicked on and why.  In fact, I have never heard anyone admit they even look at the “Pay-per-click” ads – like the ones on the right-hand-side in Google – but clearly there are lots of people who click on those. The participants aren’t lying – they genuinely don’t understand how they absorb messages and act on them.  Instead, ask if they attend any specific conferences, if they consult with Gartner before making purchases, and other more direct questions.

Your market research firm can be invaluable in guiding you to ask questions that yield actionable results.  Let their experience help you make the best use of the valuable time you have with participants.