Category — In-depth Interviews

I admit it.  Although web surveys are one of our most popular research offerings, I strongly prefer qualitative research (focus groups, in-depth interviews).

I can’t help looking at the findings in web surveys and feel like I want to dig in and ask more questions.  The participants say things that don’t make sense to me and I want to know why.  Or they make short comments responding to open-ended questions that leave me with a dozen follow-on questions I don’t have the opportunity to ask.

When I present qualitative findings, I know I can answer any question that comes up with complete confidence.  With quantitative findings, I always know that there will be questions where the answer is, “We can’t draw any conclusions based on this survey.”

This came up again last earlier this month:  We just did a series of in-depth interviews where we wanted to understand perceptions about the cost of various alternatives.  All the participants in this study had identified themselves as product decision makers who had full visibility into costs – a requirement for the study.   If you had just looked at the first level of answers people gave, you would have thought that one of the tools we were looking at was very inexpensive compared to other options.

But because this was an in-depth interview, I got to ask that all-important “why” question. I quickly realized that while all the participants had been educated on the “line 3” costs that were billed directly to their organization, not everyone was aware of the additional “line 10” costs that had to be added to support this different approach.  When you added both of those up, the tool that originally appeared less expensive turned out to have a similar TCO to other options.

Now, it’s true that we could have found this out by writing a good web survey, but one of the secrets to writing great web surveys is to know the answers to all the questions first.  We continue to recommend web surveys as good vehicles for quantifying concepts that you know well, but want to put an accurate % by each of the options.  This is a valuable thing to do, especially for market sizing, external marketing and PR purpose.

But for finding out the answers that you don’t really know, start with qualitative research – and by all means do a web survey next to put those %s in place once you know the statements to put the %s with.

A while back we blogged about phone surveys, and argued that they do not have a place in technology market research. Several comments were made both in the blog comments and to us directly that made us realize that there is confusion about the difference between phone surveys and in-depth interviews (also called IDIs). Here’s how we view this:

Phone surveys and in-depth interviews are similar because:

• Both are conducted over the phone.

But that’s pretty much the only similarity.

The differences between phone surveys and in-depth interviews:

Phone surveys: A quantitative research method that includes a large numbers of participants.

• Capture input to a common set of questions with pre-set answer options
• Administered by a phone survey professional that has a pleasant voice and is trained not to guide the responses of the participants in any way
• Are usually short: 5-20 minutes as rule of thumb
• Usually no incentive is given to participants, although there may be a small amount given.

In-Depth-Interviews (IDIs):A qualitative research method that uses a smaller number of highly select participants.

• Screener is written to allow significant discovery, open-ended questions, and drill-down
• Administered by a trained moderator who is versed in the client’s business, the goals of the research, the topic of study (in our case technology) as well as techniques for putting the client at ease and getting the kind of feedback desired
• Usually longer: 30 minutes to 1 hour is common
• Typically generous incentives are given to participants to compensate them for the time commitment

An easy way to think of the differences are that phone surveys are pretty much like web surveys, but conducted over the phone. Given these definitions, we stick by our earlier recommendation that phone surveys have no place in technology market research. They are significantly more expensive to conduct than web surveys without adding value in a corporate IT study (although agree that there are audiences phone surveys are appropriate for).

That said, there is nothing inherently wrong with phone-based research. In fact, in-depth telephone interviews have become one of our most valuable research methodologies, particularly as clients become more global and travel costs are becoming more of a factor in evaluating research budgets.

Scott Anthony recently advised, “In Market Research, Use Numbers with Caution.” He added, “Companies too frequently default to quantitative research because they think there is safety in numbers. It’s a lot easier to justify a strategy by saying, ‘The data suggests’ than by saying, ‘My intuition suggests.’ But sometimes numbers provide false confidence and obscure real opportunity.”

Anthony’s point of view is quite different than the point of view presented by Robb Mandelbaum, who recently said in Inc., “Given limited resources … it generally makes sense to go quantitative.”

Both articles are excellent  – and both present very different points of view.

So how do you choose between quantitative (surveys) and qualitative (focus groups, in-depth interviews) when performing market research projects with technology participants?

Sometimes, quantitative (surveys) research is better

1. When you need data to support a claim with investors, press, or internal stakeholders.  Dimensional Research has done a number of Web surveys that our customers have used for PR purposes, including these recent ones on anti-virus and desktop power management.

2. For trending purposes, quantitative studies are also the best.  Dimensional Research has a number of clients that follow the “Would you recommend this company to a friend?”  question promoted by HBR, and watch the responses to that trend over time.  It’s a great way to track trends in customer feedback.

3. Of course, if time is the greatest consideration and you need some kind of quick feedback – Web surveys have a big advantage.  The are FAST.

Sometimes,  qualitative (focus groups/in-depth interviews) research is better

Numbers can be deceiving and there is no better way to find that out than to talk live to people who give you numbers.  I was recently doing a competitive study, speaking to end users of a client’s competitor’s product.  My client, as is natural, was most interested in the negative feedback about the product.

So to introduce the topic of what was good and bad in the product, we started by asking the customers to rate the product being discussed on a scale of 1-5.  And of course, asked the important market research follow up question, “why?”

I was consistently surprised by users who had raved about the product, saying it was perfect and there was nothing they would change about it given the chance – and they’d assign a 3 out of 5.  And other users who basically spent the whole conversation whining and complaining about the product would give it a 5!

Numeric scales can be misleading and in these cases, qualitative studies such as focus groups or interviews are better.  I would always recommend qualitative studies when you are looking for thoughtful answers including:

-          Messaging validation for products that are new to the market

-          Market validation

-          Understanding objections and barriers

-          Product feedback for enterprise products (web surveys do a better job with consumer products that are simpler to understand)

Qualitative vs. quantitative is a good conversation to have with your market research provider – although of course you should understand their expertise. If you only have a hammer every problem is a nail, so expect a qualitative-focused research house to tell you to survey thousands of prospects, and someone who only does focus groups to emphasize quantitative approaches.

A good research firm will know if they don’t have a fit with your needs and will point you in a better direction.  For example, Dimensional Research does not do conjoint analysis, but we have a great partner that we can refer you to if that’s what you need.

One of the real strengths of focus groups – in person or online – is the opportunity for a bunch of people to see a live discussion, and even ask a few additional questions.  (Yes, Dimensional Research always leaves a few minutes at the end of a focus group session for the observers to ask a follow-on question or three.) 

It can be extremely powerful to expose people who work in corporate roles and don’t get out into the field – marcom managers, R&D, developers, etc. – to direct customer and prospect feedback. Often operational people (finance, legal) pick up something important by watching their target market discuss their jobs.

 Dimensional Research always encourages as many people as possible to listen into focus groups, or to watch the videos that we record when they’re done.

 However, sometimes “listening in” is a bad idea. It basically boils down to this:  If you only see part of a project, don’t assume that’s all there is. There is a reason why you conduct 8 focus groups, not just one.  Or why you conduct 25 interviews, not just 3 or 4.

Don’t let these scenarios happen to you:

• Attend two focus groups in New York and project that experience onto Chicago, Paris, Singapore, and Tokyo.
• Listen in on only one call of a 20-call interview project.

If you decide to listen in on market research, I strongly recommend the following:

1. Do read the final report and attend the presentation of the report.  You might as well enhance your limited experience with the full power of the overall project.
2. Don’t attend just one focus group or listen in on just one call!

Real life example

We recently conducted a series of 15 customer interviews about a client’s new initiative. It has been progressing for about a year and they wanted to know what messages their customers had absorbed. We spoke to 15 of their very best customers – the kind who spend lots of money every quarter, attend the user groups, and give references. It was a good study, and very helpful in finding out what parts of the new initiative were gaining traction and what parts needed even more evangelism.

There was one of the 15 interviews where the participant absolutely “got” it. He could have given the company’s pitch, including now and vision, with no problem at all.  It was delightful.  However, he was the ONLY one of the 15 participants who did that. The rest of the participants clearly struggled with some of the visionary aspects of the messaging. As luck would have it, that was the only interview that one of the project stakeholders listened in on. Unfortunately, during the report presentation he kept interrupting to talk about how the market “really got it.” We had to very strongly emphasize that the whole project needed to be considered – not just this one guy. The company had plenty of work to do to reach their entire customer base. They were not done.

The answer is “enough to represent your market.” This number varies significantly according to the type of market research you’re conducting.

Web Surveys

For Web surveys this is a pretty straightforward question to answer. You will use quantitative methods to determine a sample size.

There are standard ways to calculate statistical validity, and a very easy-to-use calculator and descrition of the underlying statistics can be found here. You can use it to determine how many people you need to respond to a quantitative study in order to get results that reflect your target population.

You will need to know the population size of your audience - the total number of people  in the group your sample represents. Even if you don’t know the exact population size, this is not really a problem. The mathematics of probability allows you to make a pretty good guess as long as you have some kind of basic idea. The number of participants needed is not linear. As your audience gets much larger, your sample size doesn’t increase very much.

For example, to get a result that is accurate 95% of the time within 5%, you need:

Qualitative Studies – In-depth Interviews and Focus Groups

But enough statistics homework for now. The question that is less straightforward to answer, and so is asked all the time: “how many participants do I need for a qualitative study?” such as focus groups or a series of in-depth interviews.

The first thing to do when answering this question is to figure out segmentation. How many types of participants do you need to represent? This can include verticals, countries, roles, years of experience, customers vs. prospects vs. competitor customers vs. partners, and so on.  The most common segmentation when working with Corporate IT is to have two groups: “technology decision-makers” and “economic buyers”.

Once you figure this out, and map any overlap between these areas (for example, partners may also be end-users), you’re ready to go.

In my experience, with corporate IT you need about 8-10 participants of each “type,” with a minimum of 10 participants, to produce a valid study. The only exception is studies with competitor’s customers, where you typically need more participants.

Note that most market research companies, including Dimensional Research, base pricing for qualitative work on the number of participants, so doing a good job right-sizing your project will give you the most bang for your buck.

So, to be honest- it really depends.  But I know that’s not an answer to a very pragmatic question, so let me give more details here, bearing in mind that it really does depend on your specific project. 

A typical market research project takes six weeks. This is pretty standard for a small to mid-sized project, say 10-30 in-depth interviews or 4-8 focus groups.  Larger projects take longer, of course.  And we’ll talk about Web surveys later.

Here’s a pretty typical schedule for a single stage project with one series of focus groups or in-depth interviews - excluding unforeseen “bumps” or unique requirements:

Week 1: Project go-ahead. Write and approve recruiting guide. Identify source for recruiting participants - internal or external.

Week 2-3: Recruit participants. Write and approve interview guide or moderator’s guide.

Week 4-5: Conduct research. This may take less than two weeks, depending on your goals. Four to six focus groups all in the US are usually done in just one week.

 Week 6: Write and present market research report.

WARNING: The biggest schedule slippage that happens, aside from getting all the approvals in place for the project go-ahead of course, is with identifying internal participant contacts.  Using in-house lists or asking account managers to give us their contacts for recruiting may be time consuming and the schedule should be adjusted as needed.  

Web surveys are usually faster. A typical schedule for a web survey project is three weeks:  

Week 1: Project go-ahead. Determine goals. Identify lists for participation.  Write and approve questions. Upload them to survey tool.

Week 2: Field survey.

Week 3: Close survey. Do analysis including filtering and correlating findings. Write market research report.

If the survey is being conducted as a collateral piece to support outbound marketing efforts such as PR or lead gen, add another week for copy-editing and layout.

One of my favorite market research projects is message validation. When we go and get direct feedback on high-level product messages, I always know we are going to learn something very important that will dramatically increase the results of marketing programs.  Hard ROI is easy to demonstrate when your marketing programs get twice (or more) the response from the same spend, because your message is more compelling!

So how does message validation work?  We evaluate your messages – the benefits and features that you give your PR department, your marketing programs team and your agencies – and break them into digestible chunks.  We set up one-on-one interviews with both customers and prospects (messages are experienced in isolation so this is not appropriate for focus groups).  And we walk through a systematic process of vetting the messages. 

We never ask, “What do you think?”  That doesn’t force participants to share what is really going through their heads.  You need to go deeper.  Instead of “what do you think?” you must ask, “what is MOST compelling?” and “what is LEAST compelling?”.  Do that for each area of messaging that you have: pain, benefits, features, positioning paragraph, etc.  Specifically prompt for good and bad  – and always ask WHY.  The “why” helps you know if you need to throw out some messages, use different words to express the same idea, re-order the importance, or segment your messaging.

What is interesting when you do this is how small changes can make a huge, measurable difference to your programs.  On the first message validation project I did in my career, we tested four main messages that the marketing team had been using for outbound emails and direct marketing with uninspiring results. The client knew that if they could get that 1-hour meeting, the clients were hooked, but the programs just weren’t getting the same level of response.

We tested the messages as part of an overall research project, and found that 3 out of the 4 messages resonated extremely well with everybody.  But that fourth message was different. Two out of our three target segments not only didn’t find it compelling, they actually found it insulting and frustrating. 

That said, it wasn’t that the fourth message was bad across the board.  The final target segment really liked the fourth message, even though it was actually alienating the majority of the market!  The client reserved that fourth message for a campaign targeted to that specific audience.

Solid, compelling messages about your product are the basis for a successful marketing program. Message validation helps you fine-tune your messaging to ensure maximum effectiveness of your communications with customers and prospects – a great cost saver.  

Bonus Tip:  If you’re looking for budget to fund a market research project, consider adding a message validation component and using marketing program budget!

When doing qualitative research, we need to decide which is right for the client: in-depth interviews, focus groups, or maybe a combination of both. This depends on the client’s goals.

 When are focus groups better?

Dimensional Research recommends focus groups when the client wants to gain multiple perspectives in an interactive group setting.

One of the main benefits of focus groups is that they get the participants brainstorming. When one participant’s comment feeds off of another comment and so on, the group can really dig deep into an issue. When trying to evaluate market acceptance, capture challenges and issues, or understand objections to new technologies or processes, the focus group dynamic is ideal.

Focus groups have another great benefit – the client can sit behind the glass or on a conference call and hear the direct, unfiltered feedback of a large number of participants with no distractions. Focus group sessions are also recorded for further observation.  If your goal is to expose the maximum number of your team to direct input from the market, this is a very efficient way to do it.

When are interviews better?

In-depth, one-on-one interviews with technology professionals can be conducted in person or over the phone. These are appropriate when the client wants to identify detailed perceptions, opinions, beliefs, and attitudes.

In-depth interviews are particularly effective in the following scenarios:

1. When the client’s goal is to capture feedback on experiences that occur in an isolated way, such as product messaging or product usability testing.
2. When there is anything sensitive about the feedback that participants may not feel comfortable sharing in front of other people. For example, if there has been a performance issue with a sales rep, a customer may not feel comfortable sharing details if they suspect the rep might be “behind the glass” and could hear him. The customer would feel much more comfortable sharing this information confidentially talking only with the researcher.

Two things that should NOT drive this decision are:

• Travel – Sometimes the decision is driven by the geographical locations of participants. If you’re doing customer research with customers located in diverse locations or if you want global representation in the groups without the cost of travel, you still have a choice. The one-on-one nature of a phone interviews is an easy option, or for the group dynamic choose online focus groups.
• Cost – An important consideration, of course, is cost. As a rule of thumb focus groups and interviews cost about the same per participant, so with the exception of travel – not a consideration for phone interviews or online focus groups – cost should not be the driving consideration in choosing the research approach.