Market Research Insights for Technology Companies

I was asked the other day about how to choose a research firm for qualitative global research projects – interviews and focus groups.

NOTE:  We work exclusively with technology, so this advice may not translate to consumer products – what do you think? Do let us know in the comments.

At Dimensional Research, we’ve recently done a number of projects in India, China, Japan, Australia, and Singapore, as well as Germany, France, UK, and other countries in Europe. We’ve also done virtual projects with participants from Peru, Brazil, Ecuador and Venezuela.

We have found that the principles are basically the same in any international country, with special caveats where English is not the first language.

Our recommendation is to prioritize deep experience in the industry rather than first looking for deep experience in a particular country. It is always possible to find local language support, but almost impossible to find the combination of both language and technical expertise. Of course, a firm has to have some sense of doing international research and know the pitfalls, but once you’ve done it in a few countries, the process translates to almost any other country.

The most typical approach is to pick the locations, and then partner with a local facility. Since recruiting works straight off the guide, people answer the questions the way the guide is written it to determine qualification. The important step is getting a really good guide that does a great job of filtering out unqualified people – if you do this, the local people can easily translate and get a good recruit.

Firms like Dimensional Research keep track of the really great research facilities in various countries through our direct experience and our research network – and since we specialize in technology, we work only with facilities that do great technology work. That evolves over time since there is a lot of consolidation in the market research facilities business, so it’s an ongoing effort.

In terms of moderation, if it is at all possible, we conduct research in English. Since English is the language of corporate IT and business technology, that is a real option in our research. While you need to give participants more time to communicate, the results are better if English is an option.

But it’s not always possible – in Japan it’s nearly impossible to do English-language research. In China, it depends on the audience – if you deal with very senior IT executives in the modern cities of China you might be able to use English, otherwise probably not. We have found some really excellent Japanese and Chinese moderators that we use on our projects who know technology really well and we coach them heavily before going into projects.

Bonus Tip: Make sure any firm you work with is aware of and embraces international culture. It really does make a difference to have experience outside the US to pull from. A big part of a researcher’s job is to put the participants at ease so they want to tell you the stuff that is a bit sensitive or embarrassing or doesn’t make them or their company look good.

During one of my first experiences with international research (I was only an observer), the moderator was an incredibly nice man who clearly believed that the US was the best country on earth. Unfortunately he didn’t quite get it that the British people he was talking to didn’t seem to agree with him. Not the best place to start a research conversation!

On a big reading “catch up” over the long weekend, I read two blogs posts that caught my attention:

1. The first was Seth Godin’s Are you doing math or arithmetic? This blog post resonated because my degree is in mathematics. [Side comment:  The best way to horrify your math professor is to refer to basic arithmetic activities like addition or fractions as “mathematics.” Mathematics has nothing to do with computation at a certain level. It’s about taking your facts (called axioms), and applying pure logic to come up with a proof. Knowing that something is unarguably true or false - that is the real beauty of mathematics!]
2. I also read Ivana Taylor’s challenging blog post about the future of market research firms. Ivana beautifully outlined the decreasing value that exists in collecting information – it’s simply too easy to do that these days.

So the question:  where is the value – or  the “mathematics” as Seth Godin would say – that clients pay research firms good money for.

The massive report that shows everything is not it. It’s not that difficult to spew the data from the information gathering phase into a document. It does provide value to the internal people who take the time to read and absorb the details and is not a step that should be skipped. But let’s face it, that part is the “arithmetic.”

There is a point in every research project where you have to take those massive amounts of information, gathered over a huge number of interactions, and figure out how to communicate what matters in a one-hour, or a 30- minute or even a 10-minute meeting.

This is where the value is found – in using the information to build a compelling story. This is the “mathematics” of research. Creating the story of reality  in a way that is understood and actionable by a wide audience.

Very easy to say. Very, very difficult to do. Extremely high value when you do it.

It seems that win/loss analysis is in the air these days. We’ve had a sharp uptick in incoming inquiries for projects analyzing why enterprise software deals are won or lost when there is a competitor in the mix.

The major factor in successful Win/Loss projects is the ability to identify stakeholders who were deeply involved in the process and are willing to talk about their experiences. Here are our top tips to get the right people:

1. Ensure anonymity – during an enterprise sales process, prospects can develop relationships with their account management team. They may be hesitant to discuss issues that happened during the sales process if they are worried that they’ll be putting someone they know in a bad light.
2. Expect more effort to secure losses – it’s easier to get people to talk about wins. Once a company has chosen a vendor to have a relationship with, they are more vested in that relationship and willing to put the time into making it work. If a customer has chosen another vendor, they will be less likely to talk to you. Plan for this when building target lists.
3. Keep it short – Senior IT executives value their time above all else. They are busy. It’s difficult to get an hour of their time, but asking for 15 or 30 minutes feels like much less of a commitment and they will be more likely to participate than if they feel it is a big and onerous conversation. Of course, once they’re on the phone, if they want to keep talking let them! But don’t start with the big ask.

A friend of mine who is not involved in the research business sent me this email yesterday.

Hi Diane.

I attended a conference this week and was sent this survey to fill out.

Did you watch the playback of the keynote session?
( )Yes. It was excellent.
( )Yes.  It was OK.
( )No, but I plan to.
( )No, and I am not interested.

What I want to know is where is the response “Yes and it sucked!”

Of course this was a bad survey question, which we’ve talked about before  (Bad Survey Design; Bad Web Survey Question).

But the broader point isn’t just that the research is bad, but that THE PARTICIPANTS KNOW! They can tell if their option isn’t on the list and they just won’t complete the survey if their opinion isn’t represented or will be annoyed that they weren’t able to be realistic in their answers.

My friend didn’t continue taking the survey.

You have a challenging recruit, but you finally found that last participant who fit the impossible guide that the client has given you. Of course, it turns out that she is available only at 6 AM your time, but you get up and get caffeinated and are ready for a great interview.

You dial. And you get it – the dreaded voicemail.

No problem. People have jobs, they run late, their last meeting hasn’t ended, there was traffic. So you wait five minutes, then try again. And again it is voicemail. You leave a message this time, and then, depending on your best practices, try again in another five minutes or send an email or all of the above.

But at some point you have to admit it – you have a no-show.

We’ve all been there. In B2B market research it’s really a fact of life. Our research participants have jobs that dictate their priorities, which means they sometimes can’t make a call. Since we work with corporate IT, we know that our participants will hang up or even walk out of a focus group if they get an alert that mission critical systems are down. After all, our participants are the heroes who keep their applications up and running so that the business can continue to operate.

So how do you manage no-shows, since the client does require you to talk to a certain number of participants in the project? There are really only two options:

1. Schedule time for rebooks: Build it into the schedule. If you have ten interviews, a week is plenty of time to talk to ten people. But put two weeks in the schedule anyway so you have time to rebook.
2. Over-recruit: If the schedule doesn’t allow rebooks, recruit more people than you actually need. This should cover no-shows, and hey, if you get lucky and everybody shows up, the client will be thrilled that you have delivered more input than you were contractually obliged to deliver.

I was delighted to participate in last week’s Market Research Twitterversity. The topic was Project Management, and I was a “guest professor” for the session on Special Considerations for B2B Research Projects.

B2B Market Research: How Is It Different?

The first step in preparing for this session was defining what made B2B market research projects different than those focusing on consumers. Many factors are very similar, but there are two important things that stand out as very unique:

1. B2B purchases are complex: Businesses have a lot of stakeholders who impact purchases – economic buyers who own the budget, end users who drive requirements, procurement, legal, and more. Decisions are often global, which adds cultural, language and timezone complexity. Understanding these behaviors and incorporating them into the project is vital to ensuring that research goals are met.
2. Participants have jobs: The reason why you want to talk to participants in B2B projects is that they work for companies. But that means your potential participant pool is busy, has other priorities, and is frequently very well paid. It’s harder to get their attention.

Managing Schedule Risk

When talking project management, a typical question is “How long do these projects take?” We’ve blogged about this topic before, and have given general guidelines. However, the key to meeting any schedule is identifying and managing timeline risks. At Twitterversity we discussed schedule risk specific to B2B projects, and gave some tips for handling it, including:

• Plan for no-shows: Participants have jobs and those always take priority. No shows are a fact of life in B2B research. Either put time in your schedule for rebooks, or over-recruit to meet project goals.
• Be realistic about customer information: Companies often have terrible information about their own customers, or there is an account manager that must give permission before you can make a call, or there are strict usage guidelines for communicating via email. Any of these things can derail a project schedule. If your project depends on customer information, have a “straight-face” conversation early on with the gatekeeper of that data to find out the real status of that information and your ability to access it.
• Understand confidentiality issues: You must find out if participants are unable or unwilling to talk about their business in front of their competitors. And this has to happen BEFORE they are all sitting together in a focus group. If you are aware of issues early, you can build the project to ensure that no participants from competitive companies cross paths.
• Capture international needs: Regulations, language and cultural differences are often important findings in a research project, but discovering those adds complexity to the project and must be managed.
• Understand verticals: Is there only one vertical that matters? How are those weighted? In B2B projects it can be particularly important to understand the importance of government, non-profit, and education since those three verticals purchase very differently. Do you include, exclude, or perhaps even over-represent those?

In spite of these extra considerations, B2B projects can be very rewarding. We are always learning something new!

Bonus Tip

For any research project – not just B2B – stakeholder management is important. We recommend Patty Azarello’s great blog. It is not research-specific, but it gives some great insights into how to manage your stakeholders.

Model Metrics, a provider of services at the cutting edge of cloud computing, recently sponsored a Dimensional Research survey on iPad and tablet adoption in the Enterprise. It was a good project with a fun topic and the chance to work with great client. It also demonstrated very clearly the importance of knowing how your audience uses vocabulary.

Consider the following two findings from the survey:

• 22% of IT professionals reported having officially deployed tablets
• 72% have tablets in use for business purposes in the enterprise

If you don’t know the language of corporate IT, this probably doesn’t make sense. How can there be iPads in use for business purposes when they haven’t been deployed yet?

The answer is in the word “officially.” One of the most interesting findings of the study is that employees are purchasing iPads or other tablets on their own, then bringing them to work, and IT has to support them. IT hasn’t deployed the iPads “officially,” but they have to support them anyway.

A great example of how words matter in market research. You can read more about the Model Metrics study here, or download the full report here.

We are nearing the end of our year living in Europe.  One of our goals for this trip was to experience the rich Roman and Medieval history in the area – most of which requires driving south. It is very quick to get around France once you are on the freeway, but freeway entrances are fairly far apart.

From our house, you can get on the freeway by driving north or south. The south entrance, we will call this Route A, is further away, but is in the right direction once you are on the freeway. Here is the Google Map from close to our house to the town of Orange.

The north freeway entrance, we will call this Route B, is closer, but is north from where we live so takes you out of the way. However, you spend more of the time driving freeway speeds.  Here it is on Google Maps.

Now here’s the thing – my husband and I do NOT agree on which route to take, and driving in the car offers plenty of time to discuss why our particular point of view is the correct one.

My husband started with the stats from Google Maps to support his point of view:

• Route A: 46.3 km,  47 minutes  (shorter and faster)
• Route B: 64.3 km, 51 minutes (longer and slower)

I pointed out that Google Maps is based on algorithms and not reality, so suggested we do some real-life measurements. The next few trips south I wrote down our departure times and arrival times at a set point on the freeway both coming and going (yes, we are that kind of couple!) using both routes:

• Route A: 46 minutes, 58 minutes, 53 minutes, 52 minutes  (faster 1 out of 4 times)
• Route B: 50 minutes, 52 minutes, 51 minutes, 49 minutes  (faster 3 out of 4 times)

Now, it should be immediately obvious that my husband is an advocate of Route A, and I prefer Route B. And we both have data to prove that we are right. My husband would point out that my measurements were invalid since on the trips where Route A was slower we ran into unusual traffic. I would point out that if we ran into unusual traffic 3 out of 4 times it probably means that the traffic is not that unusual.

Of course, we should know better.  He is an engineer and I am a mathematician. Before we started all our “fact finding,” we should have just put on the table that we are biased. I don’t like roundabouts, so getting on the freeway sooner is a way to avoid them.  My husband finds the freeway boring and prefers going through the towns.

So back to the market research topic. In this case the final result doesn’t really matter. But in any market research project, it’s important to know and confront biases at the beginning of a project, so you can ensure they are addressed. Findings must be strong enough that exceptions that support existing biases are identified and understood.

As we’ve said many times before, great research is all about:

1. Finding the right people to talk to
2. Asking them the right questions

In our last post we talked about how to use the Recruiting Guide to find the right people to talk to. This post is about the next part, asking the right questions.

Whether an Interview Guide for an in-depth one-on-one conversation or a focus group Moderator’s Guide, the process of writing the document is important. During the cycles of vetting with the client, you gain deep insight into the important subtleties of the findings.  There are few hard-and-fast rules for Interview Guides, because the topics vary, but here are a few principles that have worked for us in our research.

1) Invest the time needed to get the participant comfortable – The first question has to be easy and get the participant talking. It has to be more than name and job title or the weather. You need to set the stage for being chatty and opinionated. For our IT participants we like to ask about something happening in the industry, or trace the turns of their career. The catch – this takes time. When the Guide starts getting long and you have to cut, expect someone to suggest eliminating the intro. Don’t. Stick to your guns and spend some time on this. It pays off in a dramatically better conversation and stronger research results when people let their guard down and get chatty.

2) “Peel the Onion” – A good research guide starts more broadly and then gets progressively more specific. For example, start with challenges – what is hard about what they do or what is hard about a particular area. If the challenge the project is trying to address is mentioned unprompted, that is very interesting to know. If it’s not mentioned, go to the next layer of the onion and find out why it wasn’t mentioned. Then drill down one more layer into the actual solution being testing, and so on.

3) Use open-ended questions, but be prepared to prompt – An in-depth interview is not a phone survey. You want to give the participant lots of opportunities to tell you their thoughts on the topic and point to issues that you haven’t thought of. But sometimes people need a gentle push in the right direction, so if your open-ended question isn’t getting you anywhere, be prepared with some options that might get the participant talking. Be sure to present the options in a “let me clarify the question …” tone, not in an “is it A or B” tone.

Great research is all about:

1. Finding the right people to talk to
2. Asking them the right questions

The Recruiting Guide is the researcher’s tool to ensure we get the first one right – talk to the right people. It is an important written agreement between the researcher and the client that must be approved. This ensures that at the end of the study you know you’ve talked to the right people and helps prevent the “Why did we talk to  THAT guy?” syndrome.

But Recruiting Guides can be tricky. The core problem is that prospects don’t talk like marketers do.  Marketers like to segment their audience into tidy boxes, but in real life peoples’ roles aren’t that clean. With Recruiting Guides, it is particularly important to be careful about any questions where you ask about responsibility.

IT professionals typically have great pride of ownership in their work – which is great for the businesses that rely on IT business services to function. However, it can be difficult for researchers who need to find a certain type of person, because many people will put up their hands to say they have that responsibility.

In a recent study we were looking for IT procurement participants.  We needed to talk to the people who did the hands-on financial and contract work, not the technical buyer who determines the product to purchase and then passes that off to procurement to get the paperwork done. Definitely the way NOT to write the Recruiting Guide in this case was to ask, “Do you have responsibility for IT procurement?” Probably every employee in the IT organization would answer “yes” to that. They all evaluate technology solutions and make recommendations. They feel ownership and responsibility for procurement, even though they don’t have formal procurement roles.

Instead, we asked questions in the guide about reporting structure and the focus of their jobs – how much of their time was spent on procurement? – and we got exactly the people we wanted.

Practical Tip: Think about someone who works with the people you want in your study, but that you don’t want to participate. Imagine how they would answer the questions and pass the screener.  It can be challenging, but if you look at every question in the Recruiting Guide and think how someone might qualify for a study even though they aren’t the persona you want, the result will be a better project.