Market Research Insights for Technology Companies

You know this one, right? The market research project with a great recruit, good feedback, and actionable information. The segments you targeted were well represented, and a comprehensive view of the problem being studied was presented.

Then it happens. The curse strikes! The discussion during the report presentation focuses on just one participant: Some individual who was dynamic, funny, and perfectly represented a specific view of one of the study sponsors.

So why is that a curse? If the fascinating participant is an outlier who represents a view that is not in line with the majority of the study, then you have a problem. The opinion of one participant should never be over-weighted, no matter how articulate they are.

A couple of tips for how to handle this:

• Make sure the client observes enough participants to balance out any single point of view
• Look for cues that you have seen the “Fascinating Outlier”. Then, in the “post-research huddle” immediately put the input in perspective. If you’ve seen enough participants, strongly point out how unusual the participant’s point of view was. If it’s early in the study, set the stage for different input with a comment like “we’ll of course have to see what turns up during the rest of the study.”
• Specifically call out the outlier’s input in the report. Have a section labeled “Outlier Input” (or “Off-the-Island Input” if your audience isn’t market research savvy) that includes an analysis of the persona, why they were different, and why their feedback was not in line with the rest of the study.

Energetic, passionate, articulate research participants are great, and outlier opinions are important to balance any study. But following these tips will help put their feedback clearly in context.

A while back we blogged about phone surveys, and argued that they do not have a place in technology market research. Several comments were made both in the blog comments and to us directly that made us realize that there is confusion about the difference between phone surveys and in-depth interviews (also called IDIs). Here’s how we view this:

Phone surveys and in-depth interviews are similar because:

• Both are conducted over the phone.

But that’s pretty much the only similarity.

The differences between phone surveys and in-depth interviews:

Phone surveys: A quantitative research method that includes a large numbers of participants.

• Capture input to a common set of questions with pre-set answer options
• Administered by a phone survey professional that has a pleasant voice and is trained not to guide the responses of the participants in any way
• Are usually short: 5-20 minutes as rule of thumb
• Usually no incentive is given to participants, although there may be a small amount given.

In-Depth-Interviews (IDIs):A qualitative research method that uses a smaller number of highly select participants.

• Screener is written to allow significant discovery, open-ended questions, and drill-down
• Administered by a trained moderator who is versed in the client’s business, the goals of the research, the topic of study (in our case technology) as well as techniques for putting the client at ease and getting the kind of feedback desired
• Usually longer: 30 minutes to 1 hour is common
• Typically generous incentives are given to participants to compensate them for the time commitment

An easy way to think of the differences are that phone surveys are pretty much like web surveys, but conducted over the phone. Given these definitions, we stick by our earlier recommendation that phone surveys have no place in technology market research. They are significantly more expensive to conduct than web surveys without adding value in a corporate IT study (although agree that there are audiences phone surveys are appropriate for).

That said, there is nothing inherently wrong with phone-based research. In fact, in-depth telephone interviews have become one of our most valuable research methodologies, particularly as clients become more global and travel costs are becoming more of a factor in evaluating research budgets.

I went back and forth about writing this post in case it seemed self-serving, but it’s a question that we get a lot, so I wanted to offer some guidelines to consider when choosing a vendor for a market research project.

1) What is the firm’s expertise?
Does the firm you are dealing with know anything about your target market? At Dimensional Research, we work exclusively with technology. Our researchers have spent their entire careers working with the people, processes, and technology used by Corporate IT, and I can say very confidently that we know our stuff.

This matters for two reasons. If it’s a technology-related research project: (1) We’ll get better results than any research firm that doesn’t know technology, because we’ll understand the language being used and the content it’s being used in.  This means we do better follow up and analysis. (2) We won’t consume a lot of your team’s time with “hand-holding” while the researcher comes up to speed. We’ll dive right in and be productive immediately.

2) What is the researcher’s expertise?
Once you have found a firm that is a match, make sure you talk to the person who will be doing the actual research – conducting the interviews, moderating the focus groups, writing and analyzing the Web survey. Is that person an expert also? Not every researcher at a firm knows everything the firm has experience with. Ensure you have the right person or team of people doing the hands-on work or the experience the firm brings will be essentially irrelevant.

3) Does the firm demonstrate expertise in their own marketing?
So you think you’ve found a good market research firm? Now look critically at their marketing. This will depend to some degree on the project of course, so look at the appropriate items:

• If you have someone doing Web usability research, is their own Web site design usable?
• If you’re doing a message validation, how good is their messaging? Is it compelling? Did they understand your pain?
• If they’re doing competitive research, what do they know about their own competition and can they articulate a differentiator?
• If it’s a customer satisfaction project, how satisfied are their own customers? Do they track customer satisfaction? How?

4) Has the firm/researcher done a similar project before?
If the firm’s answer to this question is “yes,” get a reference call set up, or a very, very good reason why not to set up such a call. Researchers tend to develop very close relationships with their clients – during a project there is a lot of communication – and if you can’t get a reference, that’s a big red flag.

If the answer is “no,” that’s not necessarily a deal breaker – there is a first time for everything – but is your research firm honest about that? Are they willing to be open throughout the process so mistakes are caught early?

I’ll make a confession – the first time Dimensional Research tried using social media to recruit it didn’t go very well. We were still learning the ropes and didn’t understand where our audience was hanging out. But the client knew we were trying something new, and we had an extra two weeks built into the schedule to course correct. Good thing, because we needed it, but the end result was a very successful project.

5) THE MOST IMPORTANT ONE:  How have they evolved their own business based on their research?
I was shocked to meet a fellow researcher recently who essentially hasn’t changed their business in ten years.  If a research firm doesn’t fundamentally understand the value of research enough to use it to make their own business better, DO NOT hire them.  A good researcher always has their research hat on for their own business.  Dimensional Research treats every engagement as ongoing research into our own business to launch new products like our ROI analysis.  We also constantly test our assumptions, as in this example of what we did with stipends.

The other great thing about question #5, is that it can be answered by any firm.  One of the frustrations of market research is that the vast majority of our work is confidential, so it’s impossible to share our biggest successes.  But any firm should be able to articulate how they use research to drive their own business.

I was reading an interesting market research article by Tim McAtee. This line near the end of the article really jumped out at me: “…this approach can make research more enjoyable for the subject by making survey vehicles less boring.”

This raised a couple of questions for me:

1. How much fun do our participants have when they are engaged in a project with us?
2. Does it matter to the outcome of the research?

Fun can certainly add value. Any interviewer knows that getting someone to relax during the interview results in more honest answers and better research findings. We purposely write our interview guides to start off with more general topics so we can get the participant engaged.

Of course, you do have to cover the topic of the research well, but if the participant is qualified to talk about a topic, they probably really enjoy talking about it. And of course a completely boring survey, especially with too many ranking or rating matrix questions, will have a huge drop-off, or if the stipend is really good, they’ll speed through the survey to get the completes without giving thoughtful answers.

On the flip side you can have some bad results in a focus group when there is too much positive energy floating around the room – which is just as bad as an overly negative atmosphere. You can’t have an environment where people are hesitant to share negative viewpoints because they don’t want to bring everyone down.

IMHO there are some definitive statements about fun and market research:

• Boring is NEVER good for market research.
• Participants who enjoy the process  give better feedback.
• Fun cannot come at the expense of research goals.

I have one more theory: If the researcher is engaged and enjoying the study, that helps the outcome too. Here at Dimensional Research, we love what we do! We find the IT professionals we have the pleasure to work with endlessly fascinating. They have a truly difficult job and their creativity, tenacity, and smarts consistently impress us.

We hope that we’ve made it clear in this blog that Dimensional  Research is a big fan of Web surveys.   Web surveys are a great market research tool. They make it easy to get immediate results right at your fingertips.

However, you need to have a certain level of knowledge before you can run an effective Web survey.  Here are three signs that might indicate you are not ready for a Web survey, and you should do some qualitative research first.

1. If you write multi-choice questions and are completely guessing the answers, you’re probably not ready for a Web survey.  Of course, you always need to have an “Other” section to cover the corner-cases that you just didn’t think of, but your options must capture most of the likely responses to effectively quantify a finding.  Let’s admit it, survey takers will often pick a a presented option that isn’t quite right rather than take the time to fill in an open-ended option, so you can’t rely on “other” to cover your lack of knowledge.
2. If you’re asking a lot of open-ended questions, you’re probably not ready for a Web survey.  But what is “a lot?”  Good rule of thumb, no more than one open-ended question for every 20 survey questions – not including that important last question “Is there anything else you’d like to tell us?” that you put at the end of every survey.
3. If you get only one shot at an audience, don’t waste it with an uninformed Web survey.  If you don’t completely understand a new market, and have a participant list that you can use only once, mitigate the risk that you’ve gotten something wrong by doing a few interviews with a couple of list members, or doing some small trial surveys with more open-ended questions that you can use to form a better Web survey.

Don’t waste a Web survey. If participants are taking the time to give you feedback, make sure you’re getting the most from them.  Don’t be scared to add some test surveys or interviews to a project schedule – the extra week or two needed in the schedule will give dramatically higher results.

We just wrapped up a great project that started as a stand-alone Web survey. We needed to quantify some specific purchasing metrics, so the survey was clearly the right methodology.   But as we drilled deeper, we realized that we were making far too many guesses about actual purchase motivations.

We added a series of 15-minute customer interviews prior to the Web survey that gave us some great insights into the  scenarios that were driving behavior, and then developed a much better Web survey that gave crystal clear data about all scenarios. Most importantly, it allowed us to eliminate a group of customers that weren’t motivated by the conditions we were evaluating (although their behavior was similar) and would have skewed our data significantly if we hadn’t excluded their responses for certain questions.

Going into the final presentation, we were very glad that we’d done the interviews.  We knew our stuff cold and had the backup we needed to defend the results and ensure they were taking seriously – a must to influence the business outcome.

Incentives are a normal part of any market research project. With each project we ask ourselves: What will we give participants?  How much will it cost?  How do we sweeten the pot to ensure high participation?

Today we’d like to take a step back and ask:  Do we really need to give people stuff to make them talk to us?

Now, before anybody panics, we do believe that most of the time the answer is “yes.” When we’re reaching out to new audiences that we have absolutely no relationship with, we need something to get their attention.  And if we’re asking for something big – driving across town to attend a focus group for example – of course the stipend is important.  We’ve discussed guidelines for compensating participants in the past, and we still think that post is relevant and useful.

But that doesn’t mean that we always need a “prize.”  Especially when working with customers, the best reward might be that they see the feedback they give influencing the product roadmap or support. Sometimes just being heard is a very strong driver.

For example, we have just completed an annual customer survey for a customer we’ve been working with for years. The first 3 years we gave a typical stipend of a $25 gift card to anyone who completed the survey.   But, this year the budget was slashed, so we figured we’d try it without a stipend and see what happened.  A couple of observations on this test:

• We got the same number of completes as last year – when we did offer a stipend!  The customer base had grown by 15% so we can expect that the lack of stipend hit our response rate by that much, but we still had a very significant response rate so the results were valid.
• Responses were higher quality than in the past (as evaluated by the thoughtfulness of responses to open-ended questions).  Since nobody completed the survey just for the $25, participants were more engaged.  They took the time because they cared.
• Perhaps this wouldn’t have worked the first time? This was the fourth in a series of customer surveys, and this client has definitely responded to the findings of past surveys.  Perhaps their customers now realize the survey is worth their time for other reasons then the stipend?

Net-net: We felt we got at least as good as – and arguably better – results from the customer survey without stipends for less than half the costs of previous surveys. Moving forward, Dimensional Research is making sure we evaluate whether the stipend is truly adding value to a project.

I don’t know the AAPRO well enough to know if they were trying to be funny, but I laughed at the language they used to write this survey particular tip:  Maximize cooperation or response rates within the limits of ethical treatment of human subjects.

(Thanks to Cathy Harrison and  Bernie Malinoff for Tweeting this article.)

Medical and psychological researchers have strict ethical guidelines in place dealing with people in their research, maybe people who run Web surveys need some also.  Here is our input into guidelines that should be included in a manifesto for being kind to human participants in market research studies.

1. Never force someone to give you a wrong answer – When Steve Carrell gave Conan O’Brien his exit interview on the Tonight Show and asked if working at NBC was “great”, “really great” or “fantastic”, I had a flashback to real-life employee survey I had to take years ago.  Web surveys should always cover every possible option someone could give or offer an “other” option.  A good survey will have a comments area at the end to allow users to clarify anything they felt didn’t represent their input.
2. Use complex ranking or rating matrices with caution – It is just plain painful to answer long rating or ranking questions even when you deeply care about the subject. You always end up missing at least one line and get that oh-so-not-helpful “please answer number 14″.  Be kind and use matrices sparingly and only when there are no other approaches to get the answer you’re looking for.
3. Don’t ask questions that simply don’t matter – Seriously, does my ISP need to know if I’m a man or a woman?  Is that going to change the way that they deliver service to me?  If it does, then by all means ask, but I frequently feel like somebody took a market research 101 class back in their first year of college and ever since have opened with demographic questions on age, gender, and ethnicity independent of if it matters to the survey goals.  There is something to be said for asking “warm-up” questions that are easy to answer, but surely there is something more interesting then gender to ask.  [UPDATE:  Of course there is always the "Elinor Exception" to this rule as rightly pointed out in the comments.  If you need to add a sanity check on the quality of the sample, then go ahead and ask the gender question.]
4. Don’t expect participants to know all your lingo – Seriously.  Speak plain English  (or French or Spanish or Chinese or Japanese or whatever language your participants do).  Don’t throw in tons of jargon without defining it first.
5. Be realistic about how long your research takes – It seems as if every survey in the world is “10 minutes” long.  Surely you can have at least one person who didn’t write the survey questions try it out and tell you how long it realistically takes an average person.

I’m sure at Dimensional Research we have on occasion broken one of these rules, but we try our hardest to ensure our participants don’t suffer in the process of giving their feedback.  We’re always happiest if they actually enjoy it.

What would you add to a “Manifesto on Ethical Treatment of Survey Takers”?

It’s a bit of a Catch-22 for vendors who sell to Corporate IT. In pre-sales situations, customers demand to see evidence of ROI achieved by other customers. They want calculations and solid numbers. But then when they deploy your product they don’t track their own ROI, so you can’t share that with new prospects.

For what it’s worth, Corporate IT knows that they do this and that it isn’t fair.  I’ve talked about this with many participants in interviews and focus groups and they understand the dynamic very well.  They do appreciate your efforts to get them anything, even if it’s clearly an internal ROI that is heavily biased. But of course, they really do want that real ROI since their management is demanding it from them.

In response to direct client feedback, Dimensional Research has launched a new offering for establishing ROI, The ROI Framework. In this project, we interview your customers to get their ROI – whatever they have. Some will be solid, some will be anecdotal, but when you put together the information from 5-10 or more of these customers, it’s pretty compelling. To make it easy for customers we’ll offer anonymity so they can be completely open about their ROI and not have to jump through hoops getting corporate approvals.

We’ll take those interviews, and put together a white paper summarizing the input that can be used as a collateral piece. We will also use the content to develop an ROI calculator for use as a sales tool or on your web site. Now your ROI calculator will have as a source “Dimensional Research Customer Analysis” rather than “our best internal guess.”

Contact us today if you’re interested in helping your sales force beat the Catch-22 of ROI.

As New Year’s approaches, anyone surfing the Internet knows that it becomes increasingly hard to avoid “best of” lists. We went back and forth on whether to publish such a list, then decided that since blogs are arranged in a chronological order, highlighting the “recent” rather than the “best,” it’s sometimes difficult for readers to find the best material in a blog. This list should help – it highlights our favorite 2009 market research articles, as well as reader favorites.

Best of 2009: Reader Favorites

• In-Depth Interviews, Focus Groups, or Both? When doing qualitative research, we need to decide which is right for the client: in-depth interviews, focus groups, or maybe a combination of both. The choice depends entirely on the client’s goals.
• Top Three Questions about Competitive Research. As technology market research experts, we get a lot of inquiries about competitive research projects. This article includes answers to some of our most frequently asked questions.
• Customer Satisfaction Surveys: Avoiding “Survivor Bias”. One of the biggest mistakes in doing customer satisfaction surveys is allowing “survivor bias”.  This happens when you surveying your existing “surviving” customers – the ones that stayed with you no matter what – and ignoring the customers and business that you lost.

Best of 2009: Our Favorites

We know that these articles are relevant, because we keep sending them to clients! Among other issues, these articles answer the important question of “what is technology market research?” and explain how to measure market research ROI.

• Technology Expert Defined. Technology market experts are experts on Corporate IT and how it works. They are familiar with Corporate IT and its processes and can talk comfortably and knowledgeably about technology, processes and people in the IT industry.
• Message Validation: Market Research with Clear ROI. Hard ROI is easy to demonstrate when your marketing programs get two times, ten times or even greater response from the same spend, because your message is more compelling.
• Involve Your Customers In Market Research. Customers love being involved in market research. They don’t see it as a chore or just another thing on the “to do” list. On the contrary, they love being involved and appreciate the opportunity to voice their opinions.

These are our best, and most useful, technology market research articles for 2009. There are many more articles on this blog (over 50 articles in fact), so do feel free to browse. Some of our favorite categories are Quantitative or Qualitative and Research Questions.

Happy New Year!