Market Research Insights for Technology Companies

Incentives are a normal part of any market research project. With each project we ask ourselves: What will we give participants?  How much will it cost?  How do we sweeten the pot to ensure high participation?

Today we’d like to take a step back and ask:  Do we really need to give people stuff to make them talk to us?

Now, before anybody panics, we do believe that most of the time the answer is “yes.” When we’re reaching out to new audiences that we have absolutely no relationship with, we need something to get their attention.  And if we’re asking for something big – driving across town to attend a focus group for example – of course the stipend is important.  We’ve discussed guidelines for compensating participants in the past, and we still think that post is relevant and useful.

But that doesn’t mean that we always need a “prize.”  Especially when working with customers, the best reward might be that they see the feedback they give influencing the product roadmap or support. Sometimes just being heard is a very strong driver.

For example, we have just completed an annual customer survey for a customer we’ve been working with for years. The first 3 years we gave a typical stipend of a $25 gift card to anyone who completed the survey.   But, this year the budget was slashed, so we figured we’d try it without a stipend and see what happened.  A couple of observations on this test:

• We got the same number of completes as last year – when we did offer a stipend!  The customer base had grown by 15% so we can expect that the lack of stipend hit our response rate by that much, but we still had a very significant response rate so the results were valid.
• Responses were higher quality than in the past (as evaluated by the thoughtfulness of responses to open-ended questions).  Since nobody completed the survey just for the $25, participants were more engaged.  They took the time because they cared.
• Perhaps this wouldn’t have worked the first time? This was the fourth in a series of customer surveys, and this client has definitely responded to the findings of past surveys.  Perhaps their customers now realize the survey is worth their time for other reasons then the stipend?

Net-net: We felt we got at least as good as – and arguably better – results from the customer survey without stipends for less than half the costs of previous surveys. Moving forward, Dimensional Research is making sure we evaluate whether the stipend is truly adding value to a project.

I don’t know the AAPRO well enough to know if they were trying to be funny, but I laughed at the language they used to write this survey particular tip:  Maximize cooperation or response rates within the limits of ethical treatment of human subjects.

(Thanks to Cathy Harrison and  Bernie Malinoff for Tweeting this article.)

Medical and psychological researchers have strict ethical guidelines in place dealing with people in their research, maybe people who run Web surveys need some also.  Here is our input into guidelines that should be included in a manifesto for being kind to human participants in market research studies.

1. Never force someone to give you a wrong answer – When Steve Carrell gave Conan O’Brien his exit interview on the Tonight Show and asked if working at NBC was “great”, “really great” or “fantastic”, I had a flashback to real-life employee survey I had to take years ago.  Web surveys should always cover every possible option someone could give or offer an “other” option.  A good survey will have a comments area at the end to allow users to clarify anything they felt didn’t represent their input.
2. Use complex ranking or rating matrices with caution – It is just plain painful to answer long rating or ranking questions even when you deeply care about the subject. You always end up missing at least one line and get that oh-so-not-helpful “please answer number 14″.  Be kind and use matrices sparingly and only when there are no other approaches to get the answer you’re looking for.
3. Don’t ask questions that simply don’t matter – Seriously, does my ISP need to know if I’m a man or a woman?  Is that going to change the way that they deliver service to me?  If it does, then by all means ask, but I frequently feel like somebody took a market research 101 class back in their first year of college and ever since have opened with demographic questions on age, gender, and ethnicity independent of if it matters to the survey goals.  There is something to be said for asking “warm-up” questions that are easy to answer, but surely there is something more interesting then gender to ask.  [UPDATE:  Of course there is always the "Elinor Exception" to this rule as rightly pointed out in the comments.  If you need to add a sanity check on the quality of the sample, then go ahead and ask the gender question.]
4. Don’t expect participants to know all your lingo – Seriously.  Speak plain English  (or French or Spanish or Chinese or Japanese or whatever language your participants do).  Don’t throw in tons of jargon without defining it first.
5. Be realistic about how long your research takes – It seems as if every survey in the world is “10 minutes” long.  Surely you can have at least one person who didn’t write the survey questions try it out and tell you how long it realistically takes an average person.

I’m sure at Dimensional Research we have on occasion broken one of these rules, but we try our hardest to ensure our participants don’t suffer in the process of giving their feedback.  We’re always happiest if they actually enjoy it.

What would you add to a “Manifesto on Ethical Treatment of Survey Takers”?

It’s a bit of a Catch-22 for vendors who sell to Corporate IT. In pre-sales situations, customers demand to see evidence of ROI achieved by other customers. They want calculations and solid numbers. But then when they deploy your product they don’t track their own ROI, so you can’t share that with new prospects.

For what it’s worth, Corporate IT knows that they do this and that it isn’t fair.  I’ve talked about this with many participants in interviews and focus groups and they understand the dynamic very well.  They do appreciate your efforts to get them anything, even if it’s clearly an internal ROI that is heavily biased. But of course, they really do want that real ROI since their management is demanding it from them.

In response to direct client feedback, Dimensional Research has launched a new offering for establishing ROI, The ROI Framework. In this project, we interview your customers to get their ROI – whatever they have. Some will be solid, some will be anecdotal, but when you put together the information from 5-10 or more of these customers, it’s pretty compelling. To make it easy for customers we’ll offer anonymity so they can be completely open about their ROI and not have to jump through hoops getting corporate approvals.

We’ll take those interviews, and put together a white paper summarizing the input that can be used as a collateral piece. We will also use the content to develop an ROI calculator for use as a sales tool or on your web site. Now your ROI calculator will have as a source “Dimensional Research Customer Analysis” rather than “our best internal guess.”

Contact us today if you’re interested in helping your sales force beat the Catch-22 of ROI.

As New Year’s approaches, anyone surfing the Internet knows that it becomes increasingly hard to avoid “best of” lists. We went back and forth on whether to publish such a list, then decided that since blogs are arranged in a chronological order, highlighting the “recent” rather than the “best,” it’s sometimes difficult for readers to find the best material in a blog. This list should help – it highlights our favorite 2009 market research articles, as well as reader favorites.

Best of 2009: Reader Favorites

• In-Depth Interviews, Focus Groups, or Both? When doing qualitative research, we need to decide which is right for the client: in-depth interviews, focus groups, or maybe a combination of both. The choice depends entirely on the client’s goals.
• Top Three Questions about Competitive Research. As technology market research experts, we get a lot of inquiries about competitive research projects. This article includes answers to some of our most frequently asked questions.
• Customer Satisfaction Surveys: Avoiding “Survivor Bias”. One of the biggest mistakes in doing customer satisfaction surveys is allowing “survivor bias”.  This happens when you surveying your existing “surviving” customers – the ones that stayed with you no matter what – and ignoring the customers and business that you lost.

Best of 2009: Our Favorites

We know that these articles are relevant, because we keep sending them to clients! Among other issues, these articles answer the important question of “what is technology market research?” and explain how to measure market research ROI.

• Technology Expert Defined. Technology market experts are experts on Corporate IT and how it works. They are familiar with Corporate IT and its processes and can talk comfortably and knowledgeably about technology, processes and people in the IT industry.
• Message Validation: Market Research with Clear ROI. Hard ROI is easy to demonstrate when your marketing programs get two times, ten times or even greater response from the same spend, because your message is more compelling.
• Involve Your Customers In Market Research. Customers love being involved in market research. They don’t see it as a chore or just another thing on the “to do” list. On the contrary, they love being involved and appreciate the opportunity to voice their opinions.

These are our best, and most useful, technology market research articles for 2009. There are many more articles on this blog (over 50 articles in fact), so do feel free to browse. Some of our favorite categories are Quantitative or Qualitative and Research Questions.

Happy New Year!

One approach to market research that we haven’t talked much about in this blog is phone surveys.

Phone surveys are when a person with a nice voice calls people and asks questions from a script. Answers are recorded in a spreadsheet, and the final result is very similar to the graphs you’d expect out of a Web survey.

Phone surveys are different from in-depth interviews because the script is asked exactly as written, unlike an in-depth interview where you have a researcher who is a technology expert asking the follow-on questions needed to drill down into answers.

Dimensional Research usually doesn’t recommend phone surveys to our technology clients. Even for consumer marketing, phone surveys are becoming less useful according to Jay Leve of SurveyUSA: “There is [no] future for any form of telephone research that is predicated on the researcher being able to barge in at will and seize the respondent.”

Several more compelling reasons why phone surveys don’t work are outlined very nicely by Jeffrey Henning at Research Live. The ones that relate to market research for technology companies include:

- Expense of dialing. More and more phone surveys are done via cell phones, since more and more people (and this includes many technology startups) don’t use landlines. In the US, by law, you can only use automatic dialing for landlines, not for cellphones. Manual dialing is much more labor-intensive – and expensive.

- Online surveys eliminate the expense of data entry. The respondent to a web survey is, in effect, donating the data entry cost, as they select the appropriate choices and type in their answers. With a phone survey, you are paying a call center representative to transcribe each respondent’s replies.

- The visual medium of Web surveys lets you easily show people visual concepts, such as ads or core messages, and get their response. Surveys that require respondents to react to visual concepts can’t be conducted with phone surveys alone.

- Web surveys have the allure of confidentiality. People today feel more comfortable sharing information on the Web than answering the prying questions of a phone interviewer.

- People prefer Web surveys, because a Web survey can be done at the respondent’s convenience, rather than at the moment the phone interviewer happened to call.

Jeffrey does make a case for phone surveys in certain situations, but almost none of these are relevant to IT market research including:

• Major Account Research – In these cases we recommend in-depth interviews or online customer advisory boards. Why wouldn’t you take the opportunity to have a deep conversation with your biggest customers?
• The Human Touch – This argument only works with Corporate IT if you are also knowledgeable about technology, so again, it’s better to do in-depth interviews.
• Some People Aren’t Online – This is obviously not an issue for technology professionals.

Our recommendation, after years of doing market research with Corporate IT, is to avoid phone surveys when doing technology market research. Instead, use Web surveys, in-depth interviews, or a combination of both.

Whenever we get a chance, we love to participate in market research done by other companies. Doing so gives us the opportunity to think about how we’re answering questions about the stuff we care about. Being on the “other side of the glass” is a great reality check.

We recently upgraded some important business software that we use here at Dimensional Research, and had the opportunity to do a follow-up survey on the experience.  We quickly discovered that it was a very poorly designed survey. The company wasted their time and energy (and ours too!) on a survey that was so badly designed, it will do nothing to help make things better next time.

Here are the things they did wrong:

1) To get the product installed, you first had to register, get a new license key, download the product, then run the installer. Four distinct steps – and we had problems with the first three! But the survey asked only about the step that actually worked – the last one. There was no opportunity to tell them that we had to register three times, that we ended up phoning somebody to get our license key, and that the first site we were sent to for the download didn’t have the right file.

2) They were clearly using the same exact survey for people who were first-time buyers and for those who were upgrading. Even though we purchased three years ago and our reasons for purchasing aren’t relevant anymore, we were required to answer questions about how we chose this vendor, etc.  Even if their lists are so bad that they don’t know we were an upgrade rather than a new purchase, a better designed survey would have given us the opportunity to skip new purchaser questions and continue onto the relevant product-related questions.

3) They did not ask any product-related questions aside from the installation. We’ve been using this product for three years now and they’ve never asked us any questions about the product – only about purchasing and installing the product. It definitely left us with the impression that all they care about is getting our renewal dollars .

4) There were no open ended questions where we could inform them of our issues, so that they could fix them, or to find out if we were an isolated case or if all of their customers have had the same problems.

Today’s fabulous, easy-to-use online survey tools are enabling a lot of bad survey behavior. We trust the readers of this blog will have better judgment!

One of the realities of doing technology market research is that you end up dealing with people in the real world. For those who work in roles that deal only with the hottest new innovations, it can be a bit of a shock to shift gears from the cutting-edge of hype and the  super-early-adopters that use new technology.

There is definitely good news. Your technology market research project will give you a good dose of reality and a much better understanding of the market that you’re actually marketing and selling into. And any good market research firm will help you to find exactly the group you need to hear from: whether a cross-section of the entire market, a group of early adopters, or conservative corporate IT executives.

However, bear in mind that the market research project may not feel like the rest of your life. Everyone you talk to on a daily basis may know about your technology and your space, but that doesn’t mean everyone in the world does.

A few important things to remember:

a. “Buzz” usually isn’t happening with the entire market. It may feel like everybody is talking about cloud computing these days, but in reality they aren’t. There are plenty of smart, informed people that simply haven’t got cloud computing on their radar because they are focused on other things.

b. Your competitors are not “everywhere.” We know it feels like that to you, but in reality, only a small percentage of the market uses your competitor’s tools.

c. Even your own customers aren’t as educated about your product as you are. Don’t expect to have the same deep conversation with them that you have in your internal meetings. Remember:  you spend 120% of your time thinking about your product. Your customers probably spend only a fraction of their time doing the same.

d. In the real world, corporate IT doesn’t get as excited about change as technology startups do. It may feel like a wet blanket to hear corporate IT research participants finding the negative aspect in the amazing new technology that you know is going to change the world.  But the reality is that it’s much better to hear the objections, so you can deal with them.

You should work with your research provider to make sure that you understand exactly who you want to talk to, and it helps to be realistic about the level of effort it takes to find exactly the right people and engage them in a beneficial conversation.

In this blog, we’ve talked a lot about market research question design, and for a good reason. When preparing a market research project, it’s important to understand how you’re leading your audience. Of course you need to guide discussions and probe for specific responses, but you need to be very aware of what you’re doing and how the way you ask influences the responses.

A very simple example of a subtle guidance: When taking personality tests, if you say “take your time” you get different responses than if you say “do your best”. “Take your time” implies that you want very thoughtful answers, and people respond accordingly. “Do your best” is neutral and doesn’t have  implicit guidance built in.

Think about your research goals, and set up your questions appropriately.

If you’re doing a message test where in real life people don’t take a lot of time to think through underlying ideas, create a similar environment with guidance like “Please review and give me your first gut reaction” or “Don’t over-think this, I’m looking for your initial response.”

If you’re looking to understand pain, it may take some digging to get to the source because your participant may not actually be aware of what’s going on.  This is particularly true with technology professionals that have lived with the pain for a while and have a workaround in place. They may not notice how much time they spend doing something that is not value added because it’s just the way it works.  In that case, ask questions that encourage  thoughtfulness, such as “Why is that?” or “Really, tell me more about that” or “Could a different approach be more valuable?“

One of the great uses of market research in general, and technology market research in particular, is getting G2 on what your competitors are doing.

Market research organizations usually have much greater access to your competitors’ customers, and the anonymity of a formal market research process allows participants to be much more open.

As technology market research experts, we get a lot of inquiries about competitive research projects. Here are answers to three of our most frequently asked questions:

1. Can you find my competitor’s customers?

Yes, with only one caveat – they actually have customers!!! Getting participants for any market research project is a bit of a numbers game. There will always be some percentage of people who don’t have the time or simply aren’t interested in participating. If your competition only has 20 customers in the entire world, we probably will struggle to find them all and get them to agree to participate in a project.

But with this one caveat, we have tried-and-true methods for finding users of any particular technology. In fact, at Dimensional Research we have actually had clients with such bad internal data and complex internal processes, that it’s been easier for us to find their customers!

2. Will my competitor’s customers talk to you?

Yes, absolutely. Not only do we motivate them with appropriate compensation but people like to be heard. And technology professionals know that strong competition drives innovation, so they want their vendors to have competition, and they want to have options.

3. How specific will competitor customers get with their info?

That of course depends on the participant and how open they are, but the majority of participants will tell you everything they know. It depends to some degree on the goals of the project. If your goals are to understand the motivation for purchase – such as a win/loss project or a lost-deal analysis – those are very straightforward questions and it’s straightforward to get clear answers from customers.

If your goal is to understand weaknesses in your competitor’s product, that can also be done, but you do need to be prepared that existing customers have often worked through weaknesses, have figured out workarounds, and no longer perceive those as problems. Having some sense of what the weaknesses might be so you can encourage the participants to recall their initial response to those problems will give better results.

Remember though that this is a good news/bad news situation. It’s just as easy for YOUR competitors to talk to your customers, so at Dimensional Research we strongly recommend that you get feedback from your own users in addition to your competitors’ customers!

The other day I was talking to a client about cloud computing, and he was frustrated. He finally said, “Does ‘cloud’ stand for something in the technology, or does it stand for the lack of clarity around the concept?”

His response made me laugh, but it also reminded me about how important it is to be very clear about terms when you want people to understand you.

When I was on my high school debate team, and again later on when I studied mathematics, you always had to start by defining your terms. It was a given that you couldn’t have a discussion or prove anything unless you knew what you were talking about in the beginning.

Talking with corporate IT you have to do the same thing – especially with new technology and acronyms! When you’re talking about PAAS, SAAS,  cloud, application virtualization , ITSM, or any other jargon, be kind and give the audience a clue about what exactly you’re talking about. Never assume they already know.

And when doing technology market research, don’t guess!!! Either tell the participants exactly what you mean, or ask questions about what the participants think it means. Put in the effort to make sure you understand each other.