Market Research Insights for Technology Companies

I’m on the phone right now with my bank, doing a “post transaction” survey.  I always take these – what researcher doesn’t? And once again I’m annoyed.  What has my bank learned from doing this survey?  As far as I can tell, ABSOLUTELY NOTHING.

The purpose of my call to the bank was to get that number you need so someone can wire you money.  Let’s see how the “post transaction” survey went:

• A question about the banker I talked to.  He was just lovely.  I asked my question badly, because I forgot the name for that number (routing? IBAN? something like that), but got an answer right away.  Gave him a high number.  Good so far.
• Wait, aren’t you going to ask me about the experience of getting to my lovely banker?  Do you care that it took three calls and five minutes to get to the 30 second conversation?  Apparently not.  No questions about that.
• Now a question about the range of services offered by my banker.  No idea.  I only asked him one question.  He answered it well.  I can’t answer this question, but “don’t know“  is not an option on this survey. I can only choose “1=unacceptable through 5=excellent.”  Where’s the N/A?  Do I say “excellent” since I liked him and don’t want to get him into trouble?  What if his bonus is based on this survey?  But that will be misleading in the analysis.  I could give a 3 to at least not skew the averages much, or just hang up.  Or do I give him a 5 because I liked that he didn’t try to sell me anything else after he answered my question?   I’ll just give a 4 to split the difference.
• What?  We’re done? Where is my “If you’d like to leave any other comments please visit www.bank.com/survey” so I can tell them about the last question.  How difficult is it to have a form for follow up in this day and age?

I will stop now before I begin ranting, but once again I feel like today’s ease of implementing surveys on web sites and telephone is causing some of the worst market research that has ever happened, and in the process annoying people who might stop doing surveys that are actually well designed and will drive better business practices.

I admit it.  Although web surveys are one of our most popular research offerings, I strongly prefer qualitative research (focus groups, in-depth interviews).

I can’t help looking at the findings in web surveys and feel like I want to dig in and ask more questions.  The participants say things that don’t make sense to me and I want to know why.  Or they make short comments responding to open-ended questions that leave me with a dozen follow-on questions I don’t have the opportunity to ask.

When I present qualitative findings, I know I can answer any question that comes up with complete confidence.  With quantitative findings, I always know that there will be questions where the answer is, “We can’t draw any conclusions based on this survey.”

This came up again last earlier this month:  We just did a series of in-depth interviews where we wanted to understand perceptions about the cost of various alternatives.  All the participants in this study had identified themselves as product decision makers who had full visibility into costs – a requirement for the study.   If you had just looked at the first level of answers people gave, you would have thought that one of the tools we were looking at was very inexpensive compared to other options.

But because this was an in-depth interview, I got to ask that all-important “why” question. I quickly realized that while all the participants had been educated on the “line 3” costs that were billed directly to their organization, not everyone was aware of the additional “line 10” costs that had to be added to support this different approach.  When you added both of those up, the tool that originally appeared less expensive turned out to have a similar TCO to other options.

Now, it’s true that we could have found this out by writing a good web survey, but one of the secrets to writing great web surveys is to know the answers to all the questions first.  We continue to recommend web surveys as good vehicles for quantifying concepts that you know well, but want to put an accurate % by each of the options.  This is a valuable thing to do, especially for market sizing, external marketing and PR purpose.

But for finding out the answers that you don’t really know, start with qualitative research – and by all means do a web survey next to put those %s in place once you know the statements to put the %s with.

We’ve read with great interest Alastair Gordon’s recent article on research methods measuring emotional response.

Mr. Gordon points out that sometimes, “Don’t know” may be the real answer after all. He adds, “Many people choose brands automatically, as a matter of habit and with little thought. Reasons for attachment can get buried and forgotten. Autopilot purchase behavior prevails in lots of categories.”

Mr. Gordon is referring to consumer products, but his argument applies to technology market research as well. While “autopilot purchases” are less common in Corporate IT than in other areas, they still happen.

We recently completed a study for a client that included talking to recent purchasers of their product. The topic of the study wasn’t competition, but there were some great competitive anecdotes that popped up, so we included them in the report. After all, who doesn’t love hearing about their competitive wins?

There was one particularly interesting scenario where the research participant already owned the competitor’s product and had done a successful implementation of the it. Still, they shelled out what was for them a significant amount of money to purchase our client’s product.

When this came up during the interview, I drilled down on the reasons for not using their existing solution – missing features, total cost of ownership (even though they product was free, maybe there were support or infrastructure costs?), relationship issues with the account manager – but it really boiled down to something else entirely.

There was no “logical” reason for the purchase. It was the power of the brand – although the research participant didn’t realize that and was unable to articulate it. He “didn’t know.”

Of course, this was very good news for my client, they have a great brand that wins them competitive deals. But the participant couldn’t articulate that. It was one of those cases where “Don’t know” was the real answer after all.

First, an obvious point. We here at Dimensional Research truly believe in market research. We wouldn’t do it for a living or spend time blogging about it if we didn’t believe there was something fundamentally useful about going into the marketplace and getting feedback.

But a couple of things we saw and read this week made us pause and reflect on the value of what we do:

1. A regular feature in a popular comedy show is “New Rules.” In a recent show, one of the new rules offered was “Online shopping sites must stop asking for my feedback.” This was the biggest applause line of the entire show. The audience loved it!

2. We came across an interesting blog post from Tim Berry on Bad Research. Tim argues that it’s “Better to know what you don’t know than to make business decisions based on false information and false conclusions. If the focus group said red is better than green, nobody dares to argue for green. Even if green is really better, and the focus group was off, distorted by one very articulate and engaging green hater. Red it is.”

I understood the audience’s reaction to the “give me feedback” line. It’s frustrating to be asked to give feedback when you don’t believe it matters or don’t believe it will change anything. And I get where Tim’s angst about research comes from, especially the scenario he talks about which looks like a clear case of a “Fascinating Outlier“.

Does the research industry need their own version of the Hippocratic Oath where we swear to “first do no harm?”: We solemnly swear that we will not frustrate customers while getting their input and we will not use research to mis-inform.

We think that the learning here isn’t not to do research, it’s this: Do GOOD research.

You know this one, right? The market research project with a great recruit, good feedback, and actionable information. The segments you targeted were well represented, and a comprehensive view of the problem being studied was presented.

Then it happens. The curse strikes! The discussion during the report presentation focuses on just one participant: Some individual who was dynamic, funny, and perfectly represented a specific view of one of the study sponsors.

So why is that a curse? If the fascinating participant is an outlier who represents a view that is not in line with the majority of the study, then you have a problem. The opinion of one participant should never be over-weighted, no matter how articulate they are.

A couple of tips for how to handle this:

• Make sure the client observes enough participants to balance out any single point of view
• Look for cues that you have seen the “Fascinating Outlier”. Then, in the “post-research huddle” immediately put the input in perspective. If you’ve seen enough participants, strongly point out how unusual the participant’s point of view was. If it’s early in the study, set the stage for different input with a comment like “we’ll of course have to see what turns up during the rest of the study.”
• Specifically call out the outlier’s input in the report. Have a section labeled “Outlier Input” (or “Off-the-Island Input” if your audience isn’t market research savvy) that includes an analysis of the persona, why they were different, and why their feedback was not in line with the rest of the study.

Energetic, passionate, articulate research participants are great, and outlier opinions are important to balance any study. But following these tips will help put their feedback clearly in context.

A while back we blogged about phone surveys, and argued that they do not have a place in technology market research. Several comments were made both in the blog comments and to us directly that made us realize that there is confusion about the difference between phone surveys and in-depth interviews (also called IDIs). Here’s how we view this:

Phone surveys and in-depth interviews are similar because:

• Both are conducted over the phone.

But that’s pretty much the only similarity.

The differences between phone surveys and in-depth interviews:

Phone surveys: A quantitative research method that includes a large numbers of participants.

• Capture input to a common set of questions with pre-set answer options
• Administered by a phone survey professional that has a pleasant voice and is trained not to guide the responses of the participants in any way
• Are usually short: 5-20 minutes as rule of thumb
• Usually no incentive is given to participants, although there may be a small amount given.

In-Depth-Interviews (IDIs):A qualitative research method that uses a smaller number of highly select participants.

• Screener is written to allow significant discovery, open-ended questions, and drill-down
• Administered by a trained moderator who is versed in the client’s business, the goals of the research, the topic of study (in our case technology) as well as techniques for putting the client at ease and getting the kind of feedback desired
• Usually longer: 30 minutes to 1 hour is common
• Typically generous incentives are given to participants to compensate them for the time commitment

An easy way to think of the differences are that phone surveys are pretty much like web surveys, but conducted over the phone. Given these definitions, we stick by our earlier recommendation that phone surveys have no place in technology market research. They are significantly more expensive to conduct than web surveys without adding value in a corporate IT study (although agree that there are audiences phone surveys are appropriate for).

That said, there is nothing inherently wrong with phone-based research. In fact, in-depth telephone interviews have become one of our most valuable research methodologies, particularly as clients become more global and travel costs are becoming more of a factor in evaluating research budgets.

I went back and forth about writing this post in case it seemed self-serving, but it’s a question that we get a lot, so I wanted to offer some guidelines to consider when choosing a vendor for a market research project.

1) What is the firm’s expertise?
Does the firm you are dealing with know anything about your target market? At Dimensional Research, we work exclusively with technology. Our researchers have spent their entire careers working with the people, processes, and technology used by Corporate IT, and I can say very confidently that we know our stuff.

This matters for two reasons. If it’s a technology-related research project: (1) We’ll get better results than any research firm that doesn’t know technology, because we’ll understand the language being used and the content it’s being used in.  This means we do better follow up and analysis. (2) We won’t consume a lot of your team’s time with “hand-holding” while the researcher comes up to speed. We’ll dive right in and be productive immediately.

2) What is the researcher’s expertise?
Once you have found a firm that is a match, make sure you talk to the person who will be doing the actual research – conducting the interviews, moderating the focus groups, writing and analyzing the Web survey. Is that person an expert also? Not every researcher at a firm knows everything the firm has experience with. Ensure you have the right person or team of people doing the hands-on work or the experience the firm brings will be essentially irrelevant.

3) Does the firm demonstrate expertise in their own marketing?
So you think you’ve found a good market research firm? Now look critically at their marketing. This will depend to some degree on the project of course, so look at the appropriate items:

• If you have someone doing Web usability research, is their own Web site design usable?
• If you’re doing a message validation, how good is their messaging? Is it compelling? Did they understand your pain?
• If they’re doing competitive research, what do they know about their own competition and can they articulate a differentiator?
• If it’s a customer satisfaction project, how satisfied are their own customers? Do they track customer satisfaction? How?

4) Has the firm/researcher done a similar project before?
If the firm’s answer to this question is “yes,” get a reference call set up, or a very, very good reason why not to set up such a call. Researchers tend to develop very close relationships with their clients – during a project there is a lot of communication – and if you can’t get a reference, that’s a big red flag.

If the answer is “no,” that’s not necessarily a deal breaker – there is a first time for everything – but is your research firm honest about that? Are they willing to be open throughout the process so mistakes are caught early?

I’ll make a confession – the first time Dimensional Research tried using social media to recruit it didn’t go very well. We were still learning the ropes and didn’t understand where our audience was hanging out. But the client knew we were trying something new, and we had an extra two weeks built into the schedule to course correct. Good thing, because we needed it, but the end result was a very successful project.

5) THE MOST IMPORTANT ONE:  How have they evolved their own business based on their research?
I was shocked to meet a fellow researcher recently who essentially hasn’t changed their business in ten years.  If a research firm doesn’t fundamentally understand the value of research enough to use it to make their own business better, DO NOT hire them.  A good researcher always has their research hat on for their own business.  Dimensional Research treats every engagement as ongoing research into our own business to launch new products like our ROI analysis.  We also constantly test our assumptions, as in this example of what we did with stipends.

The other great thing about question #5, is that it can be answered by any firm.  One of the frustrations of market research is that the vast majority of our work is confidential, so it’s impossible to share our biggest successes.  But any firm should be able to articulate how they use research to drive their own business.

I was reading an interesting market research article by Tim McAtee. This line near the end of the article really jumped out at me: “…this approach can make research more enjoyable for the subject by making survey vehicles less boring.”

This raised a couple of questions for me:

1. How much fun do our participants have when they are engaged in a project with us?
2. Does it matter to the outcome of the research?

Fun can certainly add value. Any interviewer knows that getting someone to relax during the interview results in more honest answers and better research findings. We purposely write our interview guides to start off with more general topics so we can get the participant engaged.

Of course, you do have to cover the topic of the research well, but if the participant is qualified to talk about a topic, they probably really enjoy talking about it. And of course a completely boring survey, especially with too many ranking or rating matrix questions, will have a huge drop-off, or if the stipend is really good, they’ll speed through the survey to get the completes without giving thoughtful answers.

On the flip side you can have some bad results in a focus group when there is too much positive energy floating around the room – which is just as bad as an overly negative atmosphere. You can’t have an environment where people are hesitant to share negative viewpoints because they don’t want to bring everyone down.

IMHO there are some definitive statements about fun and market research:

• Boring is NEVER good for market research.
• Participants who enjoy the process  give better feedback.
• Fun cannot come at the expense of research goals.

I have one more theory: If the researcher is engaged and enjoying the study, that helps the outcome too. Here at Dimensional Research, we love what we do! We find the IT professionals we have the pleasure to work with endlessly fascinating. They have a truly difficult job and their creativity, tenacity, and smarts consistently impress us.

We hope that we’ve made it clear in this blog that Dimensional  Research is a big fan of Web surveys.   Web surveys are a great market research tool. They make it easy to get immediate results right at your fingertips.

However, you need to have a certain level of knowledge before you can run an effective Web survey.  Here are three signs that might indicate you are not ready for a Web survey, and you should do some qualitative research first.

1. If you write multi-choice questions and are completely guessing the answers, you’re probably not ready for a Web survey.  Of course, you always need to have an “Other” section to cover the corner-cases that you just didn’t think of, but your options must capture most of the likely responses to effectively quantify a finding.  Let’s admit it, survey takers will often pick a a presented option that isn’t quite right rather than take the time to fill in an open-ended option, so you can’t rely on “other” to cover your lack of knowledge.
2. If you’re asking a lot of open-ended questions, you’re probably not ready for a Web survey.  But what is “a lot?”  Good rule of thumb, no more than one open-ended question for every 20 survey questions – not including that important last question “Is there anything else you’d like to tell us?” that you put at the end of every survey.
3. If you get only one shot at an audience, don’t waste it with an uninformed Web survey.  If you don’t completely understand a new market, and have a participant list that you can use only once, mitigate the risk that you’ve gotten something wrong by doing a few interviews with a couple of list members, or doing some small trial surveys with more open-ended questions that you can use to form a better Web survey.

Don’t waste a Web survey. If participants are taking the time to give you feedback, make sure you’re getting the most from them.  Don’t be scared to add some test surveys or interviews to a project schedule – the extra week or two needed in the schedule will give dramatically higher results.

We just wrapped up a great project that started as a stand-alone Web survey. We needed to quantify some specific purchasing metrics, so the survey was clearly the right methodology.   But as we drilled deeper, we realized that we were making far too many guesses about actual purchase motivations.

We added a series of 15-minute customer interviews prior to the Web survey that gave us some great insights into the  scenarios that were driving behavior, and then developed a much better Web survey that gave crystal clear data about all scenarios. Most importantly, it allowed us to eliminate a group of customers that weren’t motivated by the conditions we were evaluating (although their behavior was similar) and would have skewed our data significantly if we hadn’t excluded their responses for certain questions.

Going into the final presentation, we were very glad that we’d done the interviews.  We knew our stuff cold and had the backup we needed to defend the results and ensure they were taking seriously – a must to influence the business outcome.