Market Research Insights for Technology Companies

One of the biggest mistakes in doing customer satisfaction surveys is allowing “survivor bias”.  This happens when you surveying your existing “surviving” customers – the ones that stayed with you no matter what – and ignoring the customers and business that you lost.

“Survivor bias” refers to drawing conclusions only from data that is available or convenient and thus systematically biasing your results. In technology market research, it means neglecting former customers and only drawing conclusions from existing customers – which are by definition happy customers (or at least happy enough that they have not stopped doing business with you).

If you really want to understand customer satisfaction, don’t just get feedback from the people who continue to be your customers no matter what you do to them. You need to include your former customers – those who have turned to your competitor or just plain don’t use the solution anymore.

Allowing survivor bias is a particularly heinous research mistake because it’s not hard to avoid.   The data is there after all – if you have data on your customers, you have data on your former customers.

This raises one very important question – do you even know that they are “former?”  The research can help you find that out, and even more importantly, find out why they are no longer customers. If there’s a problem with your solution, you need to know about it, and the sooner the better.

In this year’s annual customer satisfaction survey, make an effort to include former customers. After all, as we said here before, your goal is to get honest feedback, and when it comes to market research, negative feedback is good for you!

I was really surprised to be pointed to this article from Zoomerang (our main online survey tool provider, who I think makes a great survey product). The article states that more than 40% of online IT panels contain survey takers who are not really in IT.

Perhaps I shouldn’t be surprised.

It’s challenging to find appropriate IT participants for research – online or in person.  It requires a very specific knowledge of the people, processes, and technology in IT.  As this article clearly points out, generic market research providers don’t do well here.  I do genuinely think they have TRIED to “fix” this, but I know it’s hard and don’t believe that they really have.

The challenge is that IT professionals are compensated well  for participating in technology market research, so the motivation to misrepresent skills is high.  A better way ensure that you’re talking to real decision makers in corporate IT is to work with a provider who specializes in technology and has years of experience building up the resources to find these people, and has the technical savvy to know immediately if something has gone wrong with the recruit.

Dimensional Research specializes in technology market research, and work extensively with companies that sell to corporate IT.  That’s all we do. And because of that focus, we do it well.  We know the reliable sources to find participants that match your needs.  Just as importantly, we understand technology ourselves,  so we can quickly catch someone who is BS-ing.

Scott Anthony recently advised, “In Market Research, Use Numbers with Caution.” He added, “Companies too frequently default to quantitative research because they think there is safety in numbers. It’s a lot easier to justify a strategy by saying, ‘The data suggests’ than by saying, ‘My intuition suggests.’ But sometimes numbers provide false confidence and obscure real opportunity.”

Anthony’s point of view is quite different than the point of view presented by Robb Mandelbaum, who recently said in Inc., “Given limited resources … it generally makes sense to go quantitative.”

Both articles are excellent  – and both present very different points of view.

So how do you choose between quantitative (surveys) and qualitative (focus groups, in-depth interviews) when performing market research projects with technology participants?

Sometimes, quantitative (surveys) research is better

1. When you need data to support a claim with investors, press, or internal stakeholders.  Dimensional Research has done a number of Web surveys that our customers have used for PR purposes, including these recent ones on anti-virus and desktop power management.

2. For trending purposes, quantitative studies are also the best.  Dimensional Research has a number of clients that follow the “Would you recommend this company to a friend?”  question promoted by HBR, and watch the responses to that trend over time.  It’s a great way to track trends in customer feedback.

3. Of course, if time is the greatest consideration and you need some kind of quick feedback – Web surveys have a big advantage.  The are FAST.

Sometimes,  qualitative (focus groups/in-depth interviews) research is better

Numbers can be deceiving and there is no better way to find that out than to talk live to people who give you numbers.  I was recently doing a competitive study, speaking to end users of a client’s competitor’s product.  My client, as is natural, was most interested in the negative feedback about the product.

So to introduce the topic of what was good and bad in the product, we started by asking the customers to rate the product being discussed on a scale of 1-5.  And of course, asked the important market research follow up question, “why?”

I was consistently surprised by users who had raved about the product, saying it was perfect and there was nothing they would change about it given the chance – and they’d assign a 3 out of 5.  And other users who basically spent the whole conversation whining and complaining about the product would give it a 5!

Numeric scales can be misleading and in these cases, qualitative studies such as focus groups or interviews are better.  I would always recommend qualitative studies when you are looking for thoughtful answers including:

-          Messaging validation for products that are new to the market

-          Market validation

-          Understanding objections and barriers

-          Product feedback for enterprise products (web surveys do a better job with consumer products that are simpler to understand)

Qualitative vs. quantitative is a good conversation to have with your market research provider – although of course you should understand their expertise. If you only have a hammer every problem is a nail, so expect a qualitative-focused research house to tell you to survey thousands of prospects, and someone who only does focus groups to emphasize quantitative approaches.

A good research firm will know if they don’t have a fit with your needs and will point you in a better direction.  For example, Dimensional Research does not do conjoint analysis, but we have a great partner that we can refer you to if that’s what you need.

Hal Varian, chief economist at Google recently said that the “the sexy job in the next 10 years will be statisticians”.

I think he’s right.  Every participant in a first-world economy should have a solid understanding of stats.  Not that I’m recommending that everyone become a statistician, but with the volume of information out there, it’s important to understand how data can be used to sway you.

Here are two “statistics” about my own life:

i) I was the only female graduate in Pure Mathematics at my university in my year  (true).

ii) 50% of my graduating class who majored in Pure Mathematics were women (also true).

One of those statements paints a picture of a mathematics education that is oppressive to women, somehow subtly driving females away.  The other paints a picture of a very progressive math department that gave women the same opportunities as men.

As I’m sure you’ve figured out, there were only two people in my graduating class that majored in Pure Mathematics.  But without that information, I could have easily mislead you. Clearly the statistical significance is not there.

When someone presents information, whether a market researcher or anyone else, always think about the reliability. What’s the methodology?  How big is the data set? Is the audience who completed the research the right audience to comment on the topic?  Is it representative?

Research bias is very real. You need to be aware of this and always ask the right questions in order to determine whether information presented to you is valid – whether selling enterprise software or listening to the media.

If you’re not confident that your anti-virus software is keeping you safe, you’re not alone.

Dimensional Research recently completed a study on anti-virus and anti-malware software, sponsored by CoreTrace.  The 226 IT professionals who completed the Web survey reported that  Corporate IT believes the threat from malware is increasing, but they don’t have confidence in existing blacklisting approaches to protect them.  Key findings include:

• 80% say threat from malware is increasing
• 74% do not have confidence in blacklisting anti-malware
• 66% concerned that blacklisting anti-malware is not effective on “day-zero” of a new attack
• 50% concerned about the impact of performance scans
• 80% say the idea whitelisting is compelling, but only 9% report using whitelisting approaches to anti-malware

You can find press coverage of the report from:

• DarkReading
• eWeek Security Watch

A full copy of the report is available for download here.

Dimensional Research completed a new study on Desktop Power Management, sponsored by KACE. I found this study particularly interesting because so much of the conversation about power management has talked about power management in the data center, with very little discussion about the impact of leaving desktop computers, monitors, and laptops powered on when not in use.

The study reveals an interesting opportunity here since most participants (93%) think desktop power management can reduce costs, but only 10% are using a commercially purchased solution to do this. The press has been covering this report, with some of my favorite stories here:

• InformationWeek’s bMighty (an interesting take for SMBs)
• ZDnet
• eWeek Europe

You can download a copy of the full report from: www.kace.com/resources/Desktop-Power-Management.

This blog post can be summed into one sentence: “I talked to some people I know” is NOT market research!

Of course you should talk to people you know about your ideas, but you need to ask yourself, who is your target market?

Will your drinking buddies ever buy your enterprise software?  Maybe they will, and count yourself fortunate if you play poker with only CIOs. But you probably interact with a lot of other people who are not actually in your target market.

It’s a good idea to get ideas from everyone, but you should put significantly more emphasis on feedback from the people who are actually part of the community you target.

It may be obvious that your high school buddy who runs his family’s (very successfull) car dealership doesn’t know enough about technology to give you feedback.

The really problematic conversations are usually the ones with people who are in the periphery of your target, just not IN it – people who sell to IT in other companies (especially ones with big established brands!), VCs who invest in tech companies, journalists and bloggers – even your fellow co-workers.

All of these people will have insights for you, and you should certainly pay attention to them, but you should never use the info you get from these people INSTEAD of having conversations with your actual target market, the people who will eventually buy your solution. They are the ones you really need to talk to.

If your day-to-day routine does not easily facilitate those conversations – make a point to make it happen.  Find those people and talk to them in a way that gets unbiased feedback. Any market research firm would be happy to help.

At Dimensional Resaerch, we are big fans of social media. We blog here.  You can follow us on Twitter @DimensionalR. We’d love it if you fanned us on Facebook.

So what is the place of social media in market research? 

This is a topic that is constantly evolving as social media changes, but let me make one important observation about research bias. 

I think Twitter is an AMAZING way to get feedback from people who are on Twitter. The ways to interact and test are absolutely revolutionary and should be evaluated as a potential component of any company’s arsenal of feedback mechanisms. Now the question is - is my target market on Twitter? 

If you sell to corporate IT, I’m going to make a bold claim: Twitter is NOT the best way to get feedback from your target market! I’d argue that the people spending money on enterprise technology are particularly under-represented on Twitter.   (Now all the Tweeps can argue with me – please do!)

Using Twitter is a great way to reach the part of your target market that is on Twitter.  Maybe your early adopters are on Twitter.  And you may sell a solution that is more likely to be discussed on social media.  But caveats aside, think carefully about how your overall target market can be reached for feedback.

Bottom line: Know where your audience can be reached.  Execute market research that represents your entire audience – both the portion of it that is active on social media and the part that isn’t.

One of the real strengths of focus groups – in person or online – is the opportunity for a bunch of people to see a live discussion, and even ask a few additional questions.  (Yes, Dimensional Research always leaves a few minutes at the end of a focus group session for the observers to ask a follow-on question or three.) 

It can be extremely powerful to expose people who work in corporate roles and don’t get out into the field – marcom managers, R&D, developers, etc. – to direct customer and prospect feedback. Often operational people (finance, legal) pick up something important by watching their target market discuss their jobs.

 Dimensional Research always encourages as many people as possible to listen into focus groups, or to watch the videos that we record when they’re done.

 However, sometimes “listening in” is a bad idea. It basically boils down to this:  If you only see part of a project, don’t assume that’s all there is. There is a reason why you conduct 8 focus groups, not just one.  Or why you conduct 25 interviews, not just 3 or 4.

Don’t let these scenarios happen to you:

• Attend two focus groups in New York and project that experience onto Chicago, Paris, Singapore, and Tokyo.
• Listen in on only one call of a 20-call interview project.

If you decide to listen in on market research, I strongly recommend the following:

1. Do read the final report and attend the presentation of the report.  You might as well enhance your limited experience with the full power of the overall project.
2. Don’t attend just one focus group or listen in on just one call!

Real life example

We recently conducted a series of 15 customer interviews about a client’s new initiative. It has been progressing for about a year and they wanted to know what messages their customers had absorbed. We spoke to 15 of their very best customers – the kind who spend lots of money every quarter, attend the user groups, and give references. It was a good study, and very helpful in finding out what parts of the new initiative were gaining traction and what parts needed even more evangelism.

There was one of the 15 interviews where the participant absolutely “got” it. He could have given the company’s pitch, including now and vision, with no problem at all.  It was delightful.  However, he was the ONLY one of the 15 participants who did that. The rest of the participants clearly struggled with some of the visionary aspects of the messaging. As luck would have it, that was the only interview that one of the project stakeholders listened in on. Unfortunately, during the report presentation he kept interrupting to talk about how the market “really got it.” We had to very strongly emphasize that the whole project needed to be considered – not just this one guy. The company had plenty of work to do to reach their entire customer base. They were not done.

Focus groups are ideal when you want to gain multiple perspectives in an interactive group setting.

However, one of the things that can happen in focus groups is “group think.” The first person starts off on a tangent – good or bad – and then the entire room goes along with that first opinion. This is a dynamic that every good researcher is aware of. 

So does groupthink invalidate a focus group? Of course not, and in fact it can be very useful as long as you know it’s happening and stop it when necessary. Here are a few tips:

1. First, establish if group think is a good or a bad thing for your project.  Depending on your goals, group think can be a very bad thing. For example, in message testing you’re usually evaluating the kinds of pain/benefit statements that someone experiences in isolation. Groupthink is very damaging in that kind of environment because you end up seeing only one participant’s uneducated, unguided reaction to messages.  In this case, maybe focus groups are not the right medium – in-depth interviews or a Web survey may give you better results.
2. Remember that group think can be very good.  If you’re trying to brainstorm an idea, having somebody to kick off the conversation and then see where the group goes with it is a very good thing. A smart moderator will encourage this kind of thinking for a while and see where it goes. The most creative ideas from focus groups I’ve been in did not come from one single individual. They came from the group members inspiring each other to build off of each other’s experiences and ideas and come up with something more powerful than any of the individual input. When one participants says “what if …”, and then the next participant says “cool idea, but how about this…” and then yet another participant says “or even better, you could take that idea and try…”.  Now you’re cooking!
3. Be prepared to funnel groupthink.  When you see it happening, a good moderator will challenge the group with the opposite assumption.  Prompt some “group think” around the positive ideas, and then redirect the group to talk about the negative ideas. Insist that the group gives you multiple perspectives.

Above all – don’t be scared of negative group think! It’s very good to hear every possible objection your target audience can make about your solution BEFORE you start selling, so you can be prepared with the products, messages, and objection handling you need to be successful.