Market Research Insights for Technology Companies

There is no “right” number of questions to ask in a Web survey, but here are a few guiding principles.

You can include a lot of easy questions

It takes very little thought to answer questions about your gender, age, and country of residence.  It may take a second more to consider whether you are willing to share your total household income.  And you know really fast if you have bought a particular product or have called their tech support in the past week.

Just keep the options simple. For example, listing every possible country in the world in a drop-down list is complicated so consider asking about regions instead if the project goals allow.  It also helps to put these kind of easy questions on a single page so there isn’t time spent waiting for pages to load.

Use rating and ranking questions with caution

Markettools did a great webinar a while back.  (I can’t find the link unfortunately, if someone recognizes this study please send to me and I’ll add the link).  They analyzed the behavior of thousands of survey takers to see where drop-off happened.  The glaringly obvious finding was that rating and ranking questions  (those huge matrices with lots and lots of boxes to check) make survey takers go away.

Obviously there are times when research goals demand a rating/ranking approach, but do your best to limit your use of those types of questions when another approach will work. A checkbox series may be simpler to complete even though it is actually more questions.  Minimizing the number of options in a rating/ranking question also helps.

Consider the motivation of your audience

Customers who are giving feedback on a product they use on a regular basis will have a lot of tolerance for a large number of questions and in fact may welcome the opportunity to answer many questions that will simplify their use of your product in the long term.

We’ve done very successful customer surveys that offered no reward and took 20-30 minutes to complete.  The audience was vested in the topic and was engaged.  On the other hand, we often do Web surveys to gauge attitudes of IT professionals. In this case we offer a copy of the final report. We keep these surveys to less than 20 questions with at most one rating/ranking question and one open-ended question.  It takes about 3-5 minutes for a typical IT professional to do one of those kinds of surveys, which seems to match their motivation level to get the report.

The Metric that Matters: Time to complete

If your metric for building surveys is number of questions, you can easily end up with a complicated survey that will give terrible results because it’s confusing for participants and impossible to analyze.  Instead track the time to complete the survey.

Get somebody to take the survey who has not been involved in the development but knows the topic and see how long it takes them.  You might be surprised to learn that your “3 minute survey” actually takes 15 minutes. Much better to find out before you field rather than later, when nobody completes the survey, or you get comments about survey length instead of value-added feedback.

One of the important jobs of researchers is to “translate” between the lingo of marketing people and the way that their customers talk.  It’s an obvious thing when you say it, but a good researcher makes sure they are constantly doing a sanity check and purging “lingo” from their conversations.

I was reminded again of this basic concept on a recent project.  We were hired by a client’s Web team to learn about what type of marketing assets were most compelling for a particular kind of audience. It was a great research project – a clearly defined problem, with a clearly defined target audience.

Customers don’t think like marketers. When an IT professional goes out to get information, they don’t really think too hard if it’s a Web page, a PDF,  a video, or a even a call from their vendor’s account manager. They think about getting an answer to their problem. IT professionals don’t think of “consuming assets”, many of them don’t even know what a marketing asset is.  They just think they “get information from their vendor.”

By asking questions that used their own words to talk about real experiences (for example, “The last time you purchased this type of product, did you visit the vendor’s Web site?”) or showing them examples of specific Web sites and seeing what they’d be most likely to do, we talked to them in their own words and got great, usable answers.

The final step was to translate that back into the client’s marketing speak, because of course they needed to know if they should develop a video or a white paper or an interactive assessment or….

I am a huge fan of Patty Azzarello. Her blog on business leadership is consistently full of the kind of advice that makes you change the way you work (or wish you’d had someone to tell you before you learned the hard and painful way.)

She recently posted a great article, titled “Do We Need More Data?” It’s a great question and one that market researchers are often scared to ask, because if the answer is “no”, you lose business.

She makes a very important observation: There is time for market analysis and study, and there are times when either you know the answer, or there is no more useful data to be had.

Patty is not bashing market research here, the way some bloggers often do. She merely points out that it’s worthwhile checking in to make sure that’s what your project needs before you move forward.

She also suggests a great question that I ask all my clients: “What additional data is available that’s going to help me?” Going into a research project, you should have a good idea of how you will be able to act on what you learn. If your research shows that the market has a strong need for something that is outside of the core competency of your business, you may have done something interesting, but you haven’t added value to your business.

As always, it’s about goals. And when knowing specific information about the market is going to drive better business decisions, then you do need more data, and it’s time for a market research project.

As a mother of two girls, I absolutely loved this clever post by Julie Kurd:  Fame or shame…It’s all in the industry (or the household) and her example of how the same mom that is “fabulous” to a pre-schooler is “frustrating” to a teenager.

Aside from being funny and brilliant, this post also drives home an extremely important point about market research segmentation – especially with technology purchasers.  Too often we think of segmentation as being about role, geography, or industry only.  In technology, I would argue that maturity is an even more important segment to consider.

Those same behaviors that Early Adopters love about you – your deep technical descriptions of everything, your willingness to change course on product development quickly, your frequent code releases – will drive most of your early majority and definitely all the late majority CRAZY!!!  That same behavior that you’ve been getting positive customer feedback on for years is now generating negative feedback.  Go figure?

So, the question is, can you recruit Early Adopters? It’s definitely tricky because of course people don’t identify themselves that way.  For some areas it is a straightforward question – for example, if someone hasn’t done any virtualization yet in 2010, they’re definitely late majority or laggard.  If they’ve already implemented a private cloud in production – clear early adopter or possibly even innovator.

But in most cases, it’s not that clear.  An experienced moderator/interviewer can do some important segmentation in the analysis and credit different behaviors and feedback to maturity levels, helping you find and recruit early adopters.

As a technology-centric market research firm, our competitors aren’t usually another market research firm – very few have the experience with corporate IT that we do.  Our #1 competitor is “We know our customers, we don’t need a research firm.”

We’ll be the first to say that a company does not need to hire an external agency to do market research. (I’ll pause for a moment so all the professional market researchers can catch their breath at the blasphemy.)

Back when I had my career on the “other side of the glass,” I made it a point to talk to my customers regularly.  I’m a big fan of the Crossing the Chasm model of having the product marketing manager run the beta.  I always found that while my R&D counterpart did his magic to get a new software product to actually work, I could chat up the user and find out a lot about their pain, and why they were bothering to spend the time to deal with a product that was so buggy it needed a PhD to install it.

There are many benefits to “internal research.” It doesn’t have to be formal.  Go to a trade show and give your elevator pitch to 100 people in one day and see how many get it – you will learn a LOT from the questions they ask in response to the pitch.  Invite 5 users to a 1-hour phone call and ask them a few questions and really listen to what they have to say to each other.  Take a customer out for dinner and hear what they have to say outside of the confines of the office.  You will learn an incredible amount of important information from any of these activities.  And in fact, because of your expertise, you may get even more from these exchanges than an external researcher would.

The trick is to figure out when an external perspective is going to add additional benefit.  There are three areas where external resources can add real value:

1. Just Listening

I was presenting a customer report on product ROI and the client – a really talented product marketing manager – asked, “how do you get them to tell you that stuff?”  The problem with being an internal stakeholder is that you are expected to have the answers.  Which means that any customer interaction will be an exchange of them giving you information and you dealing with their concerns.  As researchers, we can gather input, but since we don’t really know the answers to the questions, we don’t have to defend or position anything.  We have the luxury of just listening to the feedback and drilling in to really understand it, without needing to come up with the answer on the spot like an internal stakeholder is expected to.

2. Neutrality

As researchers, we have no vested interest in the outcome of a research.  We have never fought for a particular point of view.  Good researchers never develop preconceived ideas before the research is executed.  External research (conducted well of course) is absolutely the most neutral form of feedback you can get.

3. Finding Non-Customers

Researchers have access to all kinds of people that you may not – people who have never heard of your company, competitor’s customers, prospects in a new vertical industry.  If it can be profiled and there are enough people in the world that match that profile, external researchers can find them and get them to talk to us when you can’t yourself. (Actually sometimes it’s even easier for us to find customers, depending on how bad the client’s customer database is – an ongoing problem with enterprise technology companies in spite of the improvements in CRM in the last several years.)

The real result of any research project is not the final report – it’s the growth that happens as a result of business owners acting on the findings.  Too often, great research ends up shelved because there is no action – or even outright disbelief.  As researchers, it’s our job to present our findings so that they influence future decision making.  Of course, working with your stakeholders to understand any preconceived ideas heading into the research will help to frame the discussion, but you’ll also want to understand the personas that will be consuming and acting on the research.

We have found that there are three types of research consumers that need special attention. Each should be communicated with differently. Knowing which one you have in the room will help considerably with presenting your findings in a way that would lead into action rather than disbelief and resistance.

1. The Impatient Executive

This is the guy with very little time and strongly preconceived ideas.  This individual is very smart – but be careful, they are VERY confident in their view of the world.  Communicating with this person is all about the executive summary.  Make sure you get to your 3-4 top takeaways within the first 5 minutes of the meeting.  Then watch carefully for any reaction.  If your research matches their beliefs, no problem.  Present the Executive Summary, they’ll agree, and will very likely leave the room early, happy that the rest of their team is getting a good validation of what is going on.

But keep an eye out for the place in the executive summary where the eyebrows come together, the blackberry is put down, or the ever-so-slight frown appears, then make sure you spend a little extra time on that area. Put a special emphasis on the anecdotes picked up in the research for that topic.  Share the one opinion that was the outlier and then frame that as the clear exception.

2. The Data Lover

This persona is the researcher’s best friend.  They will love you and your findings, and will spend hours drilling down into the subtle details of the findings.  The trick with this persona is to make sure they don’t derail the meeting with minutia that will bore the other attendees, or even worse, prevent you from getting to the real meat of the study.  If you know you have one of these people, a pre-brief with just them is a good idea.  Offering to “Take it offline to do the deep dive in this subject” is also an option – one that will be deeply appreciated by the Impatient Executive.

3. The Skeptic

This persona simply doesn’t believe in research – any research.  Their belief in the way the world works is based on years of experience, and a fact or two isn’t going to override that. The trick to communicating with this persona is to start with the parts of the project that do match their world view.  If you can lead off with one or two things that the skeptic agrees with, they’ll have more faith in the research and be more open to hearing the next point that is out of line with their beliefs.

Of course, that still leaves you with the most challenging scenario – the meeting with a big group of people including all of these personas, plus a few more.  Our best advice – do great research so you are confident in the meeting; get that executive summary in early; ask for questions to be held until the end if you’re getting off schedule;  and then do your best to be dynamic and engaging.

I’m on the phone right now with my bank, doing a “post transaction” survey.  I always take these – what researcher doesn’t? And once again I’m annoyed.  What has my bank learned from doing this survey?  As far as I can tell, ABSOLUTELY NOTHING.

The purpose of my call to the bank was to get that number you need so someone can wire you money.  Let’s see how the “post transaction” survey went:

• A question about the banker I talked to.  He was just lovely.  I asked my question badly, because I forgot the name for that number (routing? IBAN? something like that), but got an answer right away.  Gave him a high number.  Good so far.
• Wait, aren’t you going to ask me about the experience of getting to my lovely banker?  Do you care that it took three calls and five minutes to get to the 30 second conversation?  Apparently not.  No questions about that.
• Now a question about the range of services offered by my banker.  No idea.  I only asked him one question.  He answered it well.  I can’t answer this question, but “don’t know“  is not an option on this survey. I can only choose “1=unacceptable through 5=excellent.”  Where’s the N/A?  Do I say “excellent” since I liked him and don’t want to get him into trouble?  What if his bonus is based on this survey?  But that will be misleading in the analysis.  I could give a 3 to at least not skew the averages much, or just hang up.  Or do I give him a 5 because I liked that he didn’t try to sell me anything else after he answered my question?   I’ll just give a 4 to split the difference.
• What?  We’re done? Where is my “If you’d like to leave any other comments please visit www.bank.com/survey” so I can tell them about the last question.  How difficult is it to have a form for follow up in this day and age?

I will stop now before I begin ranting, but once again I feel like today’s ease of implementing surveys on web sites and telephone is causing some of the worst market research that has ever happened, and in the process annoying people who might stop doing surveys that are actually well designed and will drive better business practices.

I admit it.  Although web surveys are one of our most popular research offerings, I strongly prefer qualitative research (focus groups, in-depth interviews).

I can’t help looking at the findings in web surveys and feel like I want to dig in and ask more questions.  The participants say things that don’t make sense to me and I want to know why.  Or they make short comments responding to open-ended questions that leave me with a dozen follow-on questions I don’t have the opportunity to ask.

When I present qualitative findings, I know I can answer any question that comes up with complete confidence.  With quantitative findings, I always know that there will be questions where the answer is, “We can’t draw any conclusions based on this survey.”

This came up again last earlier this month:  We just did a series of in-depth interviews where we wanted to understand perceptions about the cost of various alternatives.  All the participants in this study had identified themselves as product decision makers who had full visibility into costs – a requirement for the study.   If you had just looked at the first level of answers people gave, you would have thought that one of the tools we were looking at was very inexpensive compared to other options.

But because this was an in-depth interview, I got to ask that all-important “why” question. I quickly realized that while all the participants had been educated on the “line 3” costs that were billed directly to their organization, not everyone was aware of the additional “line 10” costs that had to be added to support this different approach.  When you added both of those up, the tool that originally appeared less expensive turned out to have a similar TCO to other options.

Now, it’s true that we could have found this out by writing a good web survey, but one of the secrets to writing great web surveys is to know the answers to all the questions first.  We continue to recommend web surveys as good vehicles for quantifying concepts that you know well, but want to put an accurate % by each of the options.  This is a valuable thing to do, especially for market sizing, external marketing and PR purpose.

But for finding out the answers that you don’t really know, start with qualitative research – and by all means do a web survey next to put those %s in place once you know the statements to put the %s with.

We’ve read with great interest Alastair Gordon’s recent article on research methods measuring emotional response.

Mr. Gordon points out that sometimes, “Don’t know” may be the real answer after all. He adds, “Many people choose brands automatically, as a matter of habit and with little thought. Reasons for attachment can get buried and forgotten. Autopilot purchase behavior prevails in lots of categories.”

Mr. Gordon is referring to consumer products, but his argument applies to technology market research as well. While “autopilot purchases” are less common in Corporate IT than in other areas, they still happen.

We recently completed a study for a client that included talking to recent purchasers of their product. The topic of the study wasn’t competition, but there were some great competitive anecdotes that popped up, so we included them in the report. After all, who doesn’t love hearing about their competitive wins?

There was one particularly interesting scenario where the research participant already owned the competitor’s product and had done a successful implementation of the it. Still, they shelled out what was for them a significant amount of money to purchase our client’s product.

When this came up during the interview, I drilled down on the reasons for not using their existing solution – missing features, total cost of ownership (even though they product was free, maybe there were support or infrastructure costs?), relationship issues with the account manager – but it really boiled down to something else entirely.

There was no “logical” reason for the purchase. It was the power of the brand – although the research participant didn’t realize that and was unable to articulate it. He “didn’t know.”

Of course, this was very good news for my client, they have a great brand that wins them competitive deals. But the participant couldn’t articulate that. It was one of those cases where “Don’t know” was the real answer after all.

First, an obvious point. We here at Dimensional Research truly believe in market research. We wouldn’t do it for a living or spend time blogging about it if we didn’t believe there was something fundamentally useful about going into the marketplace and getting feedback.

But a couple of things we saw and read this week made us pause and reflect on the value of what we do:

1. A regular feature in a popular comedy show is “New Rules.” In a recent show, one of the new rules offered was “Online shopping sites must stop asking for my feedback.” This was the biggest applause line of the entire show. The audience loved it!

2. We came across an interesting blog post from Tim Berry on Bad Research. Tim argues that it’s “Better to know what you don’t know than to make business decisions based on false information and false conclusions. If the focus group said red is better than green, nobody dares to argue for green. Even if green is really better, and the focus group was off, distorted by one very articulate and engaging green hater. Red it is.”

I understood the audience’s reaction to the “give me feedback” line. It’s frustrating to be asked to give feedback when you don’t believe it matters or don’t believe it will change anything. And I get where Tim’s angst about research comes from, especially the scenario he talks about which looks like a clear case of a “Fascinating Outlier“.

Does the research industry need their own version of the Hippocratic Oath where we swear to “first do no harm?”: We solemnly swear that we will not frustrate customers while getting their input and we will not use research to mis-inform.

We think that the learning here isn’t not to do research, it’s this: Do GOOD research.