Market Research Insights for Technology Companies

One approach to market research that we haven’t talked much about in this blog is phone surveys.

Phone surveys are when a person with a nice voice calls people and asks questions from a script. Answers are recorded in a spreadsheet, and the final result is very similar to the graphs you’d expect out of a Web survey.

Phone surveys are different from in-depth interviews because the script is asked exactly as written, unlike an in-depth interview where you have a researcher who is a technology expert asking the follow-on questions needed to drill down into answers.

Dimensional Research usually doesn’t recommend phone surveys to our technology clients. Even for consumer marketing, phone surveys are becoming less useful according to Jay Leve of SurveyUSA: “There is [no] future for any form of telephone research that is predicated on the researcher being able to barge in at will and seize the respondent.”

Several more compelling reasons why phone surveys don’t work are outlined very nicely by Jeffrey Henning at Research Live. The ones that relate to market research for technology companies include:

- Expense of dialing. More and more phone surveys are done via cell phones, since more and more people (and this includes many technology startups) don’t use landlines. In the US, by law, you can only use automatic dialing for landlines, not for cellphones. Manual dialing is much more labor-intensive – and expensive.

- Online surveys eliminate the expense of data entry. The respondent to a web survey is, in effect, donating the data entry cost, as they select the appropriate choices and type in their answers. With a phone survey, you are paying a call center representative to transcribe each respondent’s replies.

- The visual medium of Web surveys lets you easily show people visual concepts, such as ads or core messages, and get their response. Surveys that require respondents to react to visual concepts can’t be conducted with phone surveys alone.

- Web surveys have the allure of confidentiality. People today feel more comfortable sharing information on the Web than answering the prying questions of a phone interviewer.

- People prefer Web surveys, because a Web survey can be done at the respondent’s convenience, rather than at the moment the phone interviewer happened to call.

Jeffrey does make a case for phone surveys in certain situations, but almost none of these are relevant to IT market research including:

• Major Account Research – In these cases we recommend in-depth interviews or online customer advisory boards. Why wouldn’t you take the opportunity to have a deep conversation with your biggest customers?
• The Human Touch – This argument only works with Corporate IT if you are also knowledgeable about technology, so again, it’s better to do in-depth interviews.
• Some People Aren’t Online – This is obviously not an issue for technology professionals.

Our recommendation, after years of doing market research with Corporate IT, is to avoid phone surveys when doing technology market research. Instead, use Web surveys, in-depth interviews, or a combination of both.

Whenever we get a chance, we love to participate in market research done by other companies. Doing so gives us the opportunity to think about how we’re answering questions about the stuff we care about. Being on the “other side of the glass” is a great reality check.

We recently upgraded some important business software that we use here at Dimensional Research, and had the opportunity to do a follow-up survey on the experience.  We quickly discovered that it was a very poorly designed survey. The company wasted their time and energy (and ours too!) on a survey that was so badly designed, it will do nothing to help make things better next time.

Here are the things they did wrong:

1) To get the product installed, you first had to register, get a new license key, download the product, then run the installer. Four distinct steps – and we had problems with the first three! But the survey asked only about the step that actually worked – the last one. There was no opportunity to tell them that we had to register three times, that we ended up phoning somebody to get our license key, and that the first site we were sent to for the download didn’t have the right file.

2) They were clearly using the same exact survey for people who were first-time buyers and for those who were upgrading. Even though we purchased three years ago and our reasons for purchasing aren’t relevant anymore, we were required to answer questions about how we chose this vendor, etc.  Even if their lists are so bad that they don’t know we were an upgrade rather than a new purchase, a better designed survey would have given us the opportunity to skip new purchaser questions and continue onto the relevant product-related questions.

3) They did not ask any product-related questions aside from the installation. We’ve been using this product for three years now and they’ve never asked us any questions about the product – only about purchasing and installing the product. It definitely left us with the impression that all they care about is getting our renewal dollars .

4) There were no open ended questions where we could inform them of our issues, so that they could fix them, or to find out if we were an isolated case or if all of their customers have had the same problems.

Today’s fabulous, easy-to-use online survey tools are enabling a lot of bad survey behavior. We trust the readers of this blog will have better judgment!

One of the realities of doing technology market research is that you end up dealing with people in the real world. For those who work in roles that deal only with the hottest new innovations, it can be a bit of a shock to shift gears from the cutting-edge of hype and the  super-early-adopters that use new technology.

There is definitely good news. Your technology market research project will give you a good dose of reality and a much better understanding of the market that you’re actually marketing and selling into. And any good market research firm will help you to find exactly the group you need to hear from: whether a cross-section of the entire market, a group of early adopters, or conservative corporate IT executives.

However, bear in mind that the market research project may not feel like the rest of your life. Everyone you talk to on a daily basis may know about your technology and your space, but that doesn’t mean everyone in the world does.

A few important things to remember:

a. “Buzz” usually isn’t happening with the entire market. It may feel like everybody is talking about cloud computing these days, but in reality they aren’t. There are plenty of smart, informed people that simply haven’t got cloud computing on their radar because they are focused on other things.

b. Your competitors are not “everywhere.” We know it feels like that to you, but in reality, only a small percentage of the market uses your competitor’s tools.

c. Even your own customers aren’t as educated about your product as you are. Don’t expect to have the same deep conversation with them that you have in your internal meetings. Remember:  you spend 120% of your time thinking about your product. Your customers probably spend only a fraction of their time doing the same.

d. In the real world, corporate IT doesn’t get as excited about change as technology startups do. It may feel like a wet blanket to hear corporate IT research participants finding the negative aspect in the amazing new technology that you know is going to change the world.  But the reality is that it’s much better to hear the objections, so you can deal with them.

You should work with your research provider to make sure that you understand exactly who you want to talk to, and it helps to be realistic about the level of effort it takes to find exactly the right people and engage them in a beneficial conversation.

In this blog, we’ve talked a lot about market research question design, and for a good reason. When preparing a market research project, it’s important to understand how you’re leading your audience. Of course you need to guide discussions and probe for specific responses, but you need to be very aware of what you’re doing and how the way you ask influences the responses.

A very simple example of a subtle guidance: When taking personality tests, if you say “take your time” you get different responses than if you say “do your best”. “Take your time” implies that you want very thoughtful answers, and people respond accordingly. “Do your best” is neutral and doesn’t have  implicit guidance built in.

Think about your research goals, and set up your questions appropriately.

If you’re doing a message test where in real life people don’t take a lot of time to think through underlying ideas, create a similar environment with guidance like “Please review and give me your first gut reaction” or “Don’t over-think this, I’m looking for your initial response.”

If you’re looking to understand pain, it may take some digging to get to the source because your participant may not actually be aware of what’s going on.  This is particularly true with technology professionals that have lived with the pain for a while and have a workaround in place. They may not notice how much time they spend doing something that is not value added because it’s just the way it works.  In that case, ask questions that encourage  thoughtfulness, such as “Why is that?” or “Really, tell me more about that” or “Could a different approach be more valuable?“

One of the great uses of market research in general, and technology market research in particular, is getting G2 on what your competitors are doing.

Market research organizations usually have much greater access to your competitors’ customers, and the anonymity of a formal market research process allows participants to be much more open.

As technology market research experts, we get a lot of inquiries about competitive research projects. Here are answers to three of our most frequently asked questions:

1. Can you find my competitor’s customers?

Yes, with only one caveat – they actually have customers!!! Getting participants for any market research project is a bit of a numbers game. There will always be some percentage of people who don’t have the time or simply aren’t interested in participating. If your competition only has 20 customers in the entire world, we probably will struggle to find them all and get them to agree to participate in a project.

But with this one caveat, we have tried-and-true methods for finding users of any particular technology. In fact, at Dimensional Research we have actually had clients with such bad internal data and complex internal processes, that it’s been easier for us to find their customers!

2. Will my competitor’s customers talk to you?

Yes, absolutely. Not only do we motivate them with appropriate compensation but people like to be heard. And technology professionals know that strong competition drives innovation, so they want their vendors to have competition, and they want to have options.

3. How specific will competitor customers get with their info?

That of course depends on the participant and how open they are, but the majority of participants will tell you everything they know. It depends to some degree on the goals of the project. If your goals are to understand the motivation for purchase – such as a win/loss project or a lost-deal analysis – those are very straightforward questions and it’s straightforward to get clear answers from customers.

If your goal is to understand weaknesses in your competitor’s product, that can also be done, but you do need to be prepared that existing customers have often worked through weaknesses, have figured out workarounds, and no longer perceive those as problems. Having some sense of what the weaknesses might be so you can encourage the participants to recall their initial response to those problems will give better results.

Remember though that this is a good news/bad news situation. It’s just as easy for YOUR competitors to talk to your customers, so at Dimensional Research we strongly recommend that you get feedback from your own users in addition to your competitors’ customers!

The other day I was talking to a client about cloud computing, and he was frustrated. He finally said, “Does ‘cloud’ stand for something in the technology, or does it stand for the lack of clarity around the concept?”

His response made me laugh, but it also reminded me about how important it is to be very clear about terms when you want people to understand you.

When I was on my high school debate team, and again later on when I studied mathematics, you always had to start by defining your terms. It was a given that you couldn’t have a discussion or prove anything unless you knew what you were talking about in the beginning.

Talking with corporate IT you have to do the same thing – especially with new technology and acronyms! When you’re talking about PAAS, SAAS,  cloud, application virtualization , ITSM, or any other jargon, be kind and give the audience a clue about what exactly you’re talking about. Never assume they already know.

And when doing technology market research, don’t guess!!! Either tell the participants exactly what you mean, or ask questions about what the participants think it means. Put in the effort to make sure you understand each other.

One of the biggest mistakes in doing customer satisfaction surveys is allowing “survivor bias”.  This happens when you surveying your existing “surviving” customers – the ones that stayed with you no matter what – and ignoring the customers and business that you lost.

“Survivor bias” refers to drawing conclusions only from data that is available or convenient and thus systematically biasing your results. In technology market research, it means neglecting former customers and only drawing conclusions from existing customers – which are by definition happy customers (or at least happy enough that they have not stopped doing business with you).

If you really want to understand customer satisfaction, don’t just get feedback from the people who continue to be your customers no matter what you do to them. You need to include your former customers – those who have turned to your competitor or just plain don’t use the solution anymore.

Allowing survivor bias is a particularly heinous research mistake because it’s not hard to avoid.   The data is there after all – if you have data on your customers, you have data on your former customers.

This raises one very important question – do you even know that they are “former?”  The research can help you find that out, and even more importantly, find out why they are no longer customers. If there’s a problem with your solution, you need to know about it, and the sooner the better.

In this year’s annual customer satisfaction survey, make an effort to include former customers. After all, as we said here before, your goal is to get honest feedback, and when it comes to market research, negative feedback is good for you!

I was really surprised to be pointed to this article from Zoomerang (our main online survey tool provider, who I think makes a great survey product). The article states that more than 40% of online IT panels contain survey takers who are not really in IT.

Perhaps I shouldn’t be surprised.

It’s challenging to find appropriate IT participants for research – online or in person.  It requires a very specific knowledge of the people, processes, and technology in IT.  As this article clearly points out, generic market research providers don’t do well here.  I do genuinely think they have TRIED to “fix” this, but I know it’s hard and don’t believe that they really have.

The challenge is that IT professionals are compensated well  for participating in technology market research, so the motivation to misrepresent skills is high.  A better way ensure that you’re talking to real decision makers in corporate IT is to work with a provider who specializes in technology and has years of experience building up the resources to find these people, and has the technical savvy to know immediately if something has gone wrong with the recruit.

Dimensional Research specializes in technology market research, and work extensively with companies that sell to corporate IT.  That’s all we do. And because of that focus, we do it well.  We know the reliable sources to find participants that match your needs.  Just as importantly, we understand technology ourselves,  so we can quickly catch someone who is BS-ing.

Scott Anthony recently advised, “In Market Research, Use Numbers with Caution.” He added, “Companies too frequently default to quantitative research because they think there is safety in numbers. It’s a lot easier to justify a strategy by saying, ‘The data suggests’ than by saying, ‘My intuition suggests.’ But sometimes numbers provide false confidence and obscure real opportunity.”

Anthony’s point of view is quite different than the point of view presented by Robb Mandelbaum, who recently said in Inc., “Given limited resources … it generally makes sense to go quantitative.”

Both articles are excellent  – and both present very different points of view.

So how do you choose between quantitative (surveys) and qualitative (focus groups, in-depth interviews) when performing market research projects with technology participants?

Sometimes, quantitative (surveys) research is better

1. When you need data to support a claim with investors, press, or internal stakeholders.  Dimensional Research has done a number of Web surveys that our customers have used for PR purposes, including these recent ones on anti-virus and desktop power management.

2. For trending purposes, quantitative studies are also the best.  Dimensional Research has a number of clients that follow the “Would you recommend this company to a friend?”  question promoted by HBR, and watch the responses to that trend over time.  It’s a great way to track trends in customer feedback.

3. Of course, if time is the greatest consideration and you need some kind of quick feedback – Web surveys have a big advantage.  The are FAST.

Sometimes,  qualitative (focus groups/in-depth interviews) research is better

Numbers can be deceiving and there is no better way to find that out than to talk live to people who give you numbers.  I was recently doing a competitive study, speaking to end users of a client’s competitor’s product.  My client, as is natural, was most interested in the negative feedback about the product.

So to introduce the topic of what was good and bad in the product, we started by asking the customers to rate the product being discussed on a scale of 1-5.  And of course, asked the important market research follow up question, “why?”

I was consistently surprised by users who had raved about the product, saying it was perfect and there was nothing they would change about it given the chance – and they’d assign a 3 out of 5.  And other users who basically spent the whole conversation whining and complaining about the product would give it a 5!

Numeric scales can be misleading and in these cases, qualitative studies such as focus groups or interviews are better.  I would always recommend qualitative studies when you are looking for thoughtful answers including:

-          Messaging validation for products that are new to the market

-          Market validation

-          Understanding objections and barriers

-          Product feedback for enterprise products (web surveys do a better job with consumer products that are simpler to understand)

Qualitative vs. quantitative is a good conversation to have with your market research provider – although of course you should understand their expertise. If you only have a hammer every problem is a nail, so expect a quantitative-focused research house to tell you to survey thousands of prospects, and someone who only does focus groups to emphasize qualitative approaches.

A good research firm will know if they don’t have a fit with your needs and will point you in a better direction.  For example, Dimensional Research does not do conjoint analysis, but we have a great partner that we can refer you to if that’s what you need.

Hal Varian, chief economist at Google recently said that the “the sexy job in the next 10 years will be statisticians”.

I think he’s right.  Every participant in a first-world economy should have a solid understanding of stats.  Not that I’m recommending that everyone become a statistician, but with the volume of information out there, it’s important to understand how data can be used to sway you.

Here are two “statistics” about my own life:

i) I was the only female graduate in Pure Mathematics at my university in my year  (true).

ii) 50% of my graduating class who majored in Pure Mathematics were women (also true).

One of those statements paints a picture of a mathematics education that is oppressive to women, somehow subtly driving females away.  The other paints a picture of a very progressive math department that gave women the same opportunities as men.

As I’m sure you’ve figured out, there were only two people in my graduating class that majored in Pure Mathematics.  But without that information, I could have easily mislead you. Clearly the statistical significance is not there.

When someone presents information, whether a market researcher or anyone else, always think about the reliability. What’s the methodology?  How big is the data set? Is the audience who completed the research the right audience to comment on the topic?  Is it representative?

Research bias is very real. You need to be aware of this and always ask the right questions in order to determine whether information presented to you is valid – whether selling enterprise software or listening to the media.