Category — Technology Market Research

We’ve read with great interest Alastair Gordon’s recent article on research methods measuring emotional response.

Mr. Gordon points out that sometimes, “Don’t know” may be the real answer after all. He adds, “Many people choose brands automatically, as a matter of habit and with little thought. Reasons for attachment can get buried and forgotten. Autopilot purchase behavior prevails in lots of categories.”

Mr. Gordon is referring to consumer products, but his argument applies to technology market research as well. While “autopilot purchases” are less common in Corporate IT than in other areas, they still happen.

We recently completed a study for a client that included talking to recent purchasers of their product. The topic of the study wasn’t competition, but there were some great competitive anecdotes that popped up, so we included them in the report. After all, who doesn’t love hearing about their competitive wins?

There was one particularly interesting scenario where the research participant already owned the competitor’s product and had done a successful implementation of the it. Still, they shelled out what was for them a significant amount of money to purchase our client’s product.

When this came up during the interview, I drilled down on the reasons for not using their existing solution – missing features, total cost of ownership (even though they product was free, maybe there were support or infrastructure costs?), relationship issues with the account manager – but it really boiled down to something else entirely.

There was no “logical” reason for the purchase. It was the power of the brand – although the research participant didn’t realize that and was unable to articulate it. He “didn’t know.”

Of course, this was very good news for my client, they have a great brand that wins them competitive deals. But the participant couldn’t articulate that. It was one of those cases where “Don’t know” was the real answer after all.

I went back and forth about writing this post in case it seemed self-serving, but it’s a question that we get a lot, so I wanted to offer some guidelines to consider when choosing a vendor for a market research project.

1) What is the firm’s expertise?
Does the firm you are dealing with know anything about your target market? At Dimensional Research, we work exclusively with technology. Our researchers have spent their entire careers working with the people, processes, and technology used by Corporate IT, and I can say very confidently that we know our stuff.

This matters for two reasons. If it’s a technology-related research project: (1) We’ll get better results than any research firm that doesn’t know technology, because we’ll understand the language being used and the content it’s being used in.  This means we do better follow up and analysis. (2) We won’t consume a lot of your team’s time with “hand-holding” while the researcher comes up to speed. We’ll dive right in and be productive immediately.

2) What is the researcher’s expertise?
Once you have found a firm that is a match, make sure you talk to the person who will be doing the actual research – conducting the interviews, moderating the focus groups, writing and analyzing the Web survey. Is that person an expert also? Not every researcher at a firm knows everything the firm has experience with. Ensure you have the right person or team of people doing the hands-on work or the experience the firm brings will be essentially irrelevant.

3) Does the firm demonstrate expertise in their own marketing?
So you think you’ve found a good market research firm? Now look critically at their marketing. This will depend to some degree on the project of course, so look at the appropriate items:

• If you have someone doing Web usability research, is their own Web site design usable?
• If you’re doing a message validation, how good is their messaging? Is it compelling? Did they understand your pain?
• If they’re doing competitive research, what do they know about their own competition and can they articulate a differentiator?
• If it’s a customer satisfaction project, how satisfied are their own customers? Do they track customer satisfaction? How?

4) Has the firm/researcher done a similar project before?
If the firm’s answer to this question is “yes,” get a reference call set up, or a very, very good reason why not to set up such a call. Researchers tend to develop very close relationships with their clients – during a project there is a lot of communication – and if you can’t get a reference, that’s a big red flag.

If the answer is “no,” that’s not necessarily a deal breaker – there is a first time for everything – but is your research firm honest about that? Are they willing to be open throughout the process so mistakes are caught early?

I’ll make a confession – the first time Dimensional Research tried using social media to recruit it didn’t go very well. We were still learning the ropes and didn’t understand where our audience was hanging out. But the client knew we were trying something new, and we had an extra two weeks built into the schedule to course correct. Good thing, because we needed it, but the end result was a very successful project.

5) THE MOST IMPORTANT ONE:  How have they evolved their own business based on their research?
I was shocked to meet a fellow researcher recently who essentially hasn’t changed their business in ten years.  If a research firm doesn’t fundamentally understand the value of research enough to use it to make their own business better, DO NOT hire them.  A good researcher always has their research hat on for their own business.  Dimensional Research treats every engagement as ongoing research into our own business to launch new products like our ROI analysis.  We also constantly test our assumptions, as in this example of what we did with stipends.

The other great thing about question #5, is that it can be answered by any firm.  One of the frustrations of market research is that the vast majority of our work is confidential, so it’s impossible to share our biggest successes.  But any firm should be able to articulate how they use research to drive their own business.

I was reading an interesting market research article by Tim McAtee. This line near the end of the article really jumped out at me: “…this approach can make research more enjoyable for the subject by making survey vehicles less boring.”

This raised a couple of questions for me:

1. How much fun do our participants have when they are engaged in a project with us?
2. Does it matter to the outcome of the research?

Fun can certainly add value. Any interviewer knows that getting someone to relax during the interview results in more honest answers and better research findings. We purposely write our interview guides to start off with more general topics so we can get the participant engaged.

Of course, you do have to cover the topic of the research well, but if the participant is qualified to talk about a topic, they probably really enjoy talking about it. And of course a completely boring survey, especially with too many ranking or rating matrix questions, will have a huge drop-off, or if the stipend is really good, they’ll speed through the survey to get the completes without giving thoughtful answers.

On the flip side you can have some bad results in a focus group when there is too much positive energy floating around the room – which is just as bad as an overly negative atmosphere. You can’t have an environment where people are hesitant to share negative viewpoints because they don’t want to bring everyone down.

IMHO there are some definitive statements about fun and market research:

• Boring is NEVER good for market research.
• Participants who enjoy the process  give better feedback.
• Fun cannot come at the expense of research goals.

I have one more theory: If the researcher is engaged and enjoying the study, that helps the outcome too. Here at Dimensional Research, we love what we do! We find the IT professionals we have the pleasure to work with endlessly fascinating. They have a truly difficult job and their creativity, tenacity, and smarts consistently impress us.

One of the realities of doing technology market research is that you end up dealing with people in the real world. For those who work in roles that deal only with the hottest new innovations, it can be a bit of a shock to shift gears from the cutting-edge of hype and the  super-early-adopters that use new technology.

There is definitely good news. Your technology market research project will give you a good dose of reality and a much better understanding of the market that you’re actually marketing and selling into. And any good market research firm will help you to find exactly the group you need to hear from: whether a cross-section of the entire market, a group of early adopters, or conservative corporate IT executives.

However, bear in mind that the market research project may not feel like the rest of your life. Everyone you talk to on a daily basis may know about your technology and your space, but that doesn’t mean everyone in the world does.

A few important things to remember:

a. “Buzz” usually isn’t happening with the entire market. It may feel like everybody is talking about cloud computing these days, but in reality they aren’t. There are plenty of smart, informed people that simply haven’t got cloud computing on their radar because they are focused on other things.

b. Your competitors are not “everywhere.” We know it feels like that to you, but in reality, only a small percentage of the market uses your competitor’s tools.

c. Even your own customers aren’t as educated about your product as you are. Don’t expect to have the same deep conversation with them that you have in your internal meetings. Remember:  you spend 120% of your time thinking about your product. Your customers probably spend only a fraction of their time doing the same.

d. In the real world, corporate IT doesn’t get as excited about change as technology startups do. It may feel like a wet blanket to hear corporate IT research participants finding the negative aspect in the amazing new technology that you know is going to change the world.  But the reality is that it’s much better to hear the objections, so you can deal with them.

You should work with your research provider to make sure that you understand exactly who you want to talk to, and it helps to be realistic about the level of effort it takes to find exactly the right people and engage them in a beneficial conversation.

The other day I was talking to a client about cloud computing, and he was frustrated. He finally said, “Does ‘cloud’ stand for something in the technology, or does it stand for the lack of clarity around the concept?”

His response made me laugh, but it also reminded me about how important it is to be very clear about terms when you want people to understand you.

When I was on my high school debate team, and again later on when I studied mathematics, you always had to start by defining your terms. It was a given that you couldn’t have a discussion or prove anything unless you knew what you were talking about in the beginning.

Talking with corporate IT you have to do the same thing – especially with new technology and acronyms! When you’re talking about PAAS, SAAS,  cloud, application virtualization , ITSM, or any other jargon, be kind and give the audience a clue about what exactly you’re talking about. Never assume they already know.

And when doing technology market research, don’t guess!!! Either tell the participants exactly what you mean, or ask questions about what the participants think it means. Put in the effort to make sure you understand each other.

I was really surprised to be pointed to this article from Zoomerang (our main online survey tool provider, who I think makes a great survey product). The article states that more than 40% of online IT panels contain survey takers who are not really in IT.

Perhaps I shouldn’t be surprised.

It’s challenging to find appropriate IT participants for research – online or in person.  It requires a very specific knowledge of the people, processes, and technology in IT.  As this article clearly points out, generic market research providers don’t do well here.  I do genuinely think they have TRIED to “fix” this, but I know it’s hard and don’t believe that they really have.

The challenge is that IT professionals are compensated well  for participating in technology market research, so the motivation to misrepresent skills is high.  A better way ensure that you’re talking to real decision makers in corporate IT is to work with a provider who specializes in technology and has years of experience building up the resources to find these people, and has the technical savvy to know immediately if something has gone wrong with the recruit.

Dimensional Research specializes in technology market research, and work extensively with companies that sell to corporate IT.  That’s all we do. And because of that focus, we do it well.  We know the reliable sources to find participants that match your needs.  Just as importantly, we understand technology ourselves,  so we can quickly catch someone who is BS-ing.

Scott Anthony recently advised, “In Market Research, Use Numbers with Caution.” He added, “Companies too frequently default to quantitative research because they think there is safety in numbers. It’s a lot easier to justify a strategy by saying, ‘The data suggests’ than by saying, ‘My intuition suggests.’ But sometimes numbers provide false confidence and obscure real opportunity.”

Anthony’s point of view is quite different than the point of view presented by Robb Mandelbaum, who recently said in Inc., “Given limited resources … it generally makes sense to go quantitative.”

Both articles are excellent  – and both present very different points of view.

So how do you choose between quantitative (surveys) and qualitative (focus groups, in-depth interviews) when performing market research projects with technology participants?

Sometimes, quantitative (surveys) research is better

1. When you need data to support a claim with investors, press, or internal stakeholders.  Dimensional Research has done a number of Web surveys that our customers have used for PR purposes, including these recent ones on anti-virus and desktop power management.

2. For trending purposes, quantitative studies are also the best.  Dimensional Research has a number of clients that follow the “Would you recommend this company to a friend?”  question promoted by HBR, and watch the responses to that trend over time.  It’s a great way to track trends in customer feedback.

3. Of course, if time is the greatest consideration and you need some kind of quick feedback – Web surveys have a big advantage.  The are FAST.

Sometimes,  qualitative (focus groups/in-depth interviews) research is better

Numbers can be deceiving and there is no better way to find that out than to talk live to people who give you numbers.  I was recently doing a competitive study, speaking to end users of a client’s competitor’s product.  My client, as is natural, was most interested in the negative feedback about the product.

So to introduce the topic of what was good and bad in the product, we started by asking the customers to rate the product being discussed on a scale of 1-5.  And of course, asked the important market research follow up question, “why?”

I was consistently surprised by users who had raved about the product, saying it was perfect and there was nothing they would change about it given the chance – and they’d assign a 3 out of 5.  And other users who basically spent the whole conversation whining and complaining about the product would give it a 5!

Numeric scales can be misleading and in these cases, qualitative studies such as focus groups or interviews are better.  I would always recommend qualitative studies when you are looking for thoughtful answers including:

-          Messaging validation for products that are new to the market

-          Market validation

-          Understanding objections and barriers

-          Product feedback for enterprise products (web surveys do a better job with consumer products that are simpler to understand)

Qualitative vs. quantitative is a good conversation to have with your market research provider – although of course you should understand their expertise. If you only have a hammer every problem is a nail, so expect a qualitative-focused research house to tell you to survey thousands of prospects, and someone who only does focus groups to emphasize quantitative approaches.

A good research firm will know if they don’t have a fit with your needs and will point you in a better direction.  For example, Dimensional Research does not do conjoint analysis, but we have a great partner that we can refer you to if that’s what you need.

This blog post can be summed into one sentence: “I talked to some people I know” is NOT market research!

Of course you should talk to people you know about your ideas, but you need to ask yourself, who is your target market?

Will your drinking buddies ever buy your enterprise software?  Maybe they will, and count yourself fortunate if you play poker with only CIOs. But you probably interact with a lot of other people who are not actually in your target market.

It’s a good idea to get ideas from everyone, but you should put significantly more emphasis on feedback from the people who are actually part of the community you target.

It may be obvious that your high school buddy who runs his family’s (very successfull) car dealership doesn’t know enough about technology to give you feedback.

The really problematic conversations are usually the ones with people who are in the periphery of your target, just not IN it – people who sell to IT in other companies (especially ones with big established brands!), VCs who invest in tech companies, journalists and bloggers – even your fellow co-workers.

All of these people will have insights for you, and you should certainly pay attention to them, but you should never use the info you get from these people INSTEAD of having conversations with your actual target market, the people who will eventually buy your solution. They are the ones you really need to talk to.

If your day-to-day routine does not easily facilitate those conversations – make a point to make it happen.  Find those people and talk to them in a way that gets unbiased feedback. Any market research firm would be happy to help.

At Dimensional Research, we are big fans of social media. We blog here.  You can follow us on Twitter @DimensionalR. We’d love it if you fanned us on Facebook.

So what is the place of social media in market research? 

This is a topic that is constantly evolving as social media changes, but let me make one important observation about research bias. 

I think Twitter is an AMAZING way to get feedback from people who are on Twitter. The ways to interact and test are absolutely revolutionary and should be evaluated as a potential component of any company’s arsenal of feedback mechanisms. Now the question is - is my target market on Twitter? 

If you sell to corporate IT, I’m going to make a bold claim: Twitter is NOT the best way to get feedback from your target market! I’d argue that the people spending money on enterprise technology are particularly under-represented on Twitter.   (Now all the Tweeps can argue with me – please do!)

Using Twitter is a great way to reach the part of your target market that is on Twitter.  Maybe your early adopters are on Twitter.  And you may sell a solution that is more likely to be discussed on social media.  But caveats aside, think carefully about how your overall target market can be reached for feedback.

Bottom line: Know where your audience can be reached.  Execute market research that represents your entire audience – both the portion of it that is active on social media and the part that isn’t.

According to a recent survey from a specialist PlayStation 3 site, geeks make the best lovers.

Let me clarify right from the start that as someone with distinct geek-ish tendencies, who is married to a confirmed geek, I have no intentions of arguing with the study’s conclusions – just with its methodology.

The problem with the survey is that it was biased. Given the audience likely to peruse a PS3-specific site, it’s fair to assume the respondents skewed more toward the “geek” end of the spectrum.

When conducting research you must know who you’re talking to in order to interpret the conclusions correctly. If they had done the same survey with a running site, would they have found out that marathoners are the best lovers?  

In technology market research, identifying your research target audience is especially important.  Customers will give you different insights than prospects, employees, partners, or even the customers of direct competitors! 

If you’re talking to your customers, you need to acknowledge that by definition they have the pain that you’re selling to, or they wouldn’t be using your product. So, you shouldn’t do a customer survey and then announce, “100% of the MARKET does this.”  Of course, it’s completely fair to say, “100% of our CUSTOMERS do this.”

In comparison, if you’re talking to the general market who isn’t as fully educated on your solution, you’ll need to tailor your questions accordingly. If you ask questions that are too detailed and specific to your solution, you’ll get uninformed answers. The type of questions that are appropriate for customers who are familiar with your product, are not the same ones to use for the general market.

For example, if you’re asking about product features, you will get vastly different answers from existing customers who know your product, and from your competitor’s customers who know an alternative approach. Both of these will be different from the answers you’ll get from prospects who don’t have any product experiences.

As with everything in market research, it comes back to goals. Your very first step should be figuring out your business goals. Then, figure out what you need to know in order to achieve these goals. Next, figure out who has that knowledge, and who doesn’t. Only once you’ve answered all these questions, you can dive into the details of the market research project.