Category — Research Questions

I don’t know the AAPRO well enough to know if they were trying to be funny, but I laughed at the language they used to write this survey particular tip:  Maximize cooperation or response rates within the limits of ethical treatment of human subjects.

(Thanks to Cathy Harrison and  Bernie Malinoff for Tweeting this article.)

Medical and psychological researchers have strict ethical guidelines in place dealing with people in their research, maybe people who run Web surveys need some also.  Here is our input into guidelines that should be included in a manifesto for being kind to human participants in market research studies.

1. Never force someone to give you a wrong answer – When Steve Carrell gave Conan O’Brien his exit interview on the Tonight Show and asked if working at NBC was “great”, “really great” or “fantastic”, I had a flashback to real-life employee survey I had to take years ago.  Web surveys should always cover every possible option someone could give or offer an “other” option.  A good survey will have a comments area at the end to allow users to clarify anything they felt didn’t represent their input.
2. Use complex ranking or rating matrices with caution – It is just plain painful to answer long rating or ranking questions even when you deeply care about the subject. You always end up missing at least one line and get that oh-so-not-helpful “please answer number 14″.  Be kind and use matrices sparingly and only when there are no other approaches to get the answer you’re looking for.
3. Don’t ask questions that simply don’t matter – Seriously, does my ISP need to know if I’m a man or a woman?  Is that going to change the way that they deliver service to me?  If it does, then by all means ask, but I frequently feel like somebody took a market research 101 class back in their first year of college and ever since have opened with demographic questions on age, gender, and ethnicity independent of if it matters to the survey goals.  There is something to be said for asking “warm-up” questions that are easy to answer, but surely there is something more interesting then gender to ask.  [UPDATE:  Of course there is always the "Elinor Exception" to this rule as rightly pointed out in the comments.  If you need to add a sanity check on the quality of the sample, then go ahead and ask the gender question.]
4. Don’t expect participants to know all your lingo – Seriously.  Speak plain English  (or French or Spanish or Chinese or Japanese or whatever language your participants do).  Don’t throw in tons of jargon without defining it first.
5. Be realistic about how long your research takes – It seems as if every survey in the world is “10 minutes” long.  Surely you can have at least one person who didn’t write the survey questions try it out and tell you how long it realistically takes an average person.

I’m sure at Dimensional Research we have on occasion broken one of these rules, but we try our hardest to ensure our participants don’t suffer in the process of giving their feedback.  We’re always happiest if they actually enjoy it.

What would you add to a “Manifesto on Ethical Treatment of Survey Takers”?

In this blog, we’ve talked a lot about market research question design, and for a good reason. When preparing a market research project, it’s important to understand how you’re leading your audience. Of course you need to guide discussions and probe for specific responses, but you need to be very aware of what you’re doing and how the way you ask influences the responses.

A very simple example of a subtle guidance: When taking personality tests, if you say “take your time” you get different responses than if you say “do your best”. “Take your time” implies that you want very thoughtful answers, and people respond accordingly. “Do your best” is neutral and doesn’t have  implicit guidance built in.

Think about your research goals, and set up your questions appropriately.

If you’re doing a message test where in real life people don’t take a lot of time to think through underlying ideas, create a similar environment with guidance like “Please review and give me your first gut reaction” or “Don’t over-think this, I’m looking for your initial response.”

If you’re looking to understand pain, it may take some digging to get to the source because your participant may not actually be aware of what’s going on.  This is particularly true with technology professionals that have lived with the pain for a while and have a workaround in place. They may not notice how much time they spend doing something that is not value added because it’s just the way it works.  In that case, ask questions that encourage  thoughtfulness, such as “Why is that?” or “Really, tell me more about that” or “Could a different approach be more valuable?“

Hal Varian, chief economist at Google recently said that the “the sexy job in the next 10 years will be statisticians”.

I think he’s right.  Every participant in a first-world economy should have a solid understanding of stats.  Not that I’m recommending that everyone become a statistician, but with the volume of information out there, it’s important to understand how data can be used to sway you.

Here are two “statistics” about my own life:

i) I was the only female graduate in Pure Mathematics at my university in my year  (true).

ii) 50% of my graduating class who majored in Pure Mathematics were women (also true).

One of those statements paints a picture of a mathematics education that is oppressive to women, somehow subtly driving females away.  The other paints a picture of a very progressive math department that gave women the same opportunities as men.

As I’m sure you’ve figured out, there were only two people in my graduating class that majored in Pure Mathematics.  But without that information, I could have easily mislead you. Clearly the statistical significance is not there.

When someone presents information, whether a market researcher or anyone else, always think about the reliability. What’s the methodology?  How big is the data set? Is the audience who completed the research the right audience to comment on the topic?  Is it representative?

Research bias is very real. You need to be aware of this and always ask the right questions in order to determine whether information presented to you is valid – whether selling enterprise software or listening to the media.

Negative feedback is good for you. 

Seriously.

While it’s very human to only want to hear how wonderful your solution is, you need to learn about any objections to your product NOW. Not later, when you have to deal with an entire slipped pipeline, because you didn’t realize there was a deal-breaker.

You need honest feedback, and you need it now.

But how do you ensure feedback that is real? Very few people, no matter how stupid they think your idea is, will actually tell you that. You have to create an environment that allows people to tell you about the downside. 

One of the more painful exercises you can go through is to put a group of people together in a focus group and ask “what DON’T you like about this solution?” Once you ask them this question, and give them permission to be brutally honest, they WILL tell you about everything they don’t like. And the list of things they don’t like will be very long. And their list will inspire the other participants to think about even more things that they don’t like.

But as I said, this is GOOD FOR YOU!  If you know about the objections, you can prepare for them in advance. In the long run, this will save you significant amounts of time and money.

According to a recent survey from a specialist PlayStation 3 site, geeks make the best lovers.

Let me clarify right from the start that as someone with distinct geek-ish tendencies, who is married to a confirmed geek, I have no intentions of arguing with the study’s conclusions – just with its methodology.

The problem with the survey is that it was biased. Given the audience likely to peruse a PS3-specific site, it’s fair to assume the respondents skewed more toward the “geek” end of the spectrum.

When conducting research you must know who you’re talking to in order to interpret the conclusions correctly. If they had done the same survey with a running site, would they have found out that marathoners are the best lovers?  

In technology market research, identifying your research target audience is especially important.  Customers will give you different insights than prospects, employees, partners, or even the customers of direct competitors! 

If you’re talking to your customers, you need to acknowledge that by definition they have the pain that you’re selling to, or they wouldn’t be using your product. So, you shouldn’t do a customer survey and then announce, “100% of the MARKET does this.”  Of course, it’s completely fair to say, “100% of our CUSTOMERS do this.”

In comparison, if you’re talking to the general market who isn’t as fully educated on your solution, you’ll need to tailor your questions accordingly. If you ask questions that are too detailed and specific to your solution, you’ll get uninformed answers. The type of questions that are appropriate for customers who are familiar with your product, are not the same ones to use for the general market.

For example, if you’re asking about product features, you will get vastly different answers from existing customers who know your product, and from your competitor’s customers who know an alternative approach. Both of these will be different from the answers you’ll get from prospects who don’t have any product experiences.

As with everything in market research, it comes back to goals. Your very first step should be figuring out your business goals. Then, figure out what you need to know in order to achieve these goals. Next, figure out who has that knowledge, and who doesn’t. Only once you’ve answered all these questions, you can dive into the details of the market research project.

So, to be honest- it really depends.  But I know that’s not an answer to a very pragmatic question, so let me give more details here, bearing in mind that it really does depend on your specific project. 

A typical market research project takes six weeks. This is pretty standard for a small to mid-sized project, say 10-30 in-depth interviews or 4-8 focus groups.  Larger projects take longer, of course.  And we’ll talk about Web surveys later.

Here’s a pretty typical schedule for a single stage project with one series of focus groups or in-depth interviews - excluding unforeseen “bumps” or unique requirements:

Week 1: Project go-ahead. Write and approve recruiting guide. Identify source for recruiting participants - internal or external.

Week 2-3: Recruit participants. Write and approve interview guide or moderator’s guide.

Week 4-5: Conduct research. This may take less than two weeks, depending on your goals. Four to six focus groups all in the US are usually done in just one week.

 Week 6: Write and present market research report.

WARNING: The biggest schedule slippage that happens, aside from getting all the approvals in place for the project go-ahead of course, is with identifying internal participant contacts.  Using in-house lists or asking account managers to give us their contacts for recruiting may be time consuming and the schedule should be adjusted as needed.  

Web surveys are usually faster. A typical schedule for a web survey project is three weeks:  

Week 1: Project go-ahead. Determine goals. Identify lists for participation.  Write and approve questions. Upload them to survey tool.

Week 2: Field survey.

Week 3: Close survey. Do analysis including filtering and correlating findings. Write market research report.

If the survey is being conducted as a collateral piece to support outbound marketing efforts such as PR or lead gen, add another week for copy-editing and layout.

One of the most important skills in moderating any market research discussion is to make sure you’re not “guiding the witness” to tell you something that is not accurate.

I’ve discussed here the importance of asking market research participants open-ended questions such as “can you tell me more about that?” and “is there anything else?”  I also written about the importance of getting honest feedback, and suggested ways to make sure you do.

Today I’d like to emphasize the importance of giving the participants permission to NOT know.

Market research participants are smart, competent people who know what is going on in their environments. It’s natural that they’re only human and don’t want to come across as unknowledgeable. Nobody wants to admit, “I don’t know.” 

However, YOU need to know the truth. If they don’t know about a product, you need to know about that and adjust your marketing efforts accordingly. It’s just as important for you to uncover that participants are unfamiliar with a product or an idea as it is to know their opinion about it.

The key is to ask questions that allow a participant to tell you the truth without feeling like an idiot.

A question that is in many research guides is an awareness question. “Are you familiar with product X?  What do you know about it?”  I don’t like asking the question this way. It almost guarantees that some selection of the participants will answer, “Yes, I’ve heard of it. I read about it in a magazine” just so they don’t have to admit they have never heard about the product.

Instead, ask “Are you familiar with product X – and please feel free to say no, that’s just as important for me to know.” This will give the participant permission to not know without feeling stupid. 

Another option is to ask, “What do you know about the following product – if anything?” that also leaves the door open for participants to admit they don’t know without worrying about disappointing you.

When you don’t listen to your customers or prospects, you risk losing them.

According to a recent report, “Did not listen to me” is the most widely experienced problem faced by 38% of services buyers. Additionally, 55% of buyers said they would be “much more likely” to consider hiring the provider “if they listened better.”  As Drew McLellan points out, if you add up the “didn’t listen” and the “talked too much,” that’s 63%.  Two thirds of your sales calls are being wasted simply because you talk too much and don’t listen enough! 

Market research is all about listening. It is also about asking the right kind of questions so that you’ll get the right kind of feedback. 

Here are three tried and true tips that I always use in my market research to make sure I actually listen to participants. These tips are applicable to every area of customer and prospect interaction.

1. Shoot for the 80/20 rule.  Always ask yourself, what percentage of the time are you talking, and what percentage of the time are the participants talking?  I shoot for the 80/20 rule.  If I’m doing more than 20% of the talking, then I’m probably not really listening.
2. Ask questions that encourage talking – not just “yes/no”.  Are you telling the person what their pain is, or are you letting them tell you about their pain?  A good researcher doesn’t say, “here is your pain – isn’t that right?”  They ask about pain to see if the pain they believe the prospect has actually comes up as a priority.  Of course you can always guide a discussion to keep it relevant, but you need to ask open questions in order to get real feedback – and give customers and prospects a real chance to say “no, I don’t care about that”.
3. Look for what you don’t know.  Never ask a participant “Do you agree?” 90% of the time they’ll say yes, because that’s the easy and polite thing to do. Your goal is to create an opportunity to hear something you DON’T know. 

To quote the Greek philosopher Epictetus, “we were born with two ears and one mouth, so we can listen twice as much as we speak.”  Thanks for the inspiration Drew.

I’ve sat through many advisory board and sales meetings with customers who have just been wined and dined. Then, during the formal part of the meeting, a product manager presents a new roadmap presentation and wraps up by asking, “what do you think?” with a face that clearly shows how proud they are of their work. 

You are very, very fortunate if you have customers that will actually give real, honest feedback in a scenario like that. They want to keep things positive and upbeat – they just saw pictures of that product manager’s kids or heard about their latest adventure trip. They don’t want to get all negative, even if that is what they really think.

One of the things that market research does really well is to intentionally CREATE an environment for honest feedback and for uncovering information you haven’t anticipated.  To create such an environment, you need to:

            1) Make sure the conversation is not about the personal relationship. Of course you want the relationship, and you want to leverage that. But the feedback can’t be influenced by whether someone likes you or not.  Of course they like you – you’re a tremendously likable person!  But the feedback must be about the topic, the market, and the pain points, not about what they think about you. 

You need to use verbal and non-verbal cues to set up the discussion so that it is open, non-judgmental, and the client is never aware that what they’re saying is bad news for you. You need to get the bad news now – not after you’ve been through a long, involved sales cycle that ends because there is “no budget,” which really means “not enough pain.”

            2) Dig for negative feedback.  Don’t just ask, “what do you think?”  Encourage people to talk about what they DON’T like.  You can’t make it better if you don’t know what’s wrong.  Ask, “What do you LIKE about what I just showed you?” then ask “What DON’T you like about what I showed you?” 

            3) Put feedback in context.  It’s not enough for a user to connect with the pain you solve – it has to be a priority!  I recently was in a focus group where a participant described a situation troubleshooting a particularly elusive problem in a complex application. It was a dramatic story with dozens of IT staff from different technology groups getting on bridge calls, pointing fingers, escalating issues, and finally getting the CIO involved in a 48-hour saga that ended up to be a rather minor configuration issue. I could feel the client behind the glass get excited because this was exactly the type of problem they envisioned their solution solving.

But then we asked how important it was to solve this kind of issue so it didn’t happen again. The participant explained that it wasn’t important at all.  It was the first time in 15 years it had happened and they had solved it.  They had moved on.  The pain wasn’t a priority.  Fortunately that group did express other pains that the client could solve, and they learned important lessons on ways of positioning their product that matched the most important pains.

Employing these techniques a great way to encourage your customers to give you real, honest feedback and to help you uncover pains you were unaware of.

Most businesses have an existing product or market that they are targeting, and their goal is to learn how to serve that marketplace more effectively. Any reasonable researcher should know that if you have a virtualization management solution, it isn’t that useful to talk about HR problems.  But while you need to stay on topic, you also don’t want to miss the chance to uncover information you are unaware of and may be a huge opportunity or a weakness you must address. This is one of the real advantages of qualitative market research where you can spend time deeply understanding the nuances of customer input.

Treat “unprompted” feedback with special importance.

Sometimes the information you don’t know that you need to ask about, is the most important information to hear.  Of course research must have a goal – but work with your researcher so they know your business and can use their judgment to understand what threads are worth digging into and what will take them into a rat hole that does not forward your business. 

The most important tool for unprompted feedback is this question:  “Is there anything else?”  This is the final question Dimensional Research uses for any market research interaction. After you’ve asked all your carefully crafted and specific questions, closing with an open-ended question can yield incredibly useful information.

Of course, this is a very important balancing act and research projects need to focus on their goals – not interesting side topics.  The core questions must be asked. But Dimensional Research makes a point of going through all input from every project with a special eye for the unprompted information.  Most of it is one-offs, with each participant having a special thing that only they are interested in.  But often you can spot trends in unprompted information that are really important. 

We’ve done projects where it turns out there was a critical development tool that needed to be integrated with the client’s product – not something they had realized was vital until the majority of participants talked about it, unprompted, in a series of customer and prospect focus groups.

In another project we didn’t realize that customer loyalty was almost non-existent, until we went through the unprompted feedback from interviews with the highly satisfied, happiest customers, and discovered they regularly evaluated the competition – not a question that we asked about specifically.

So do stay on topic.  But keep your ears open for the “+1″ opportunities or customer issues you hadn’t recognized – especially if you hear them from multiple sources.  Possibly minor product changes or messaging tweaks can help you address a different pain, target a new and lucrative market, or address important weaknesses.