Category — IT Participants

We just completed an extensive series of in-depth interviews with great IT participants from around the globe. These participants were smart and articulate. They knew their job, and were very good at it.

But as smart and capable as these participants were, we discovered during the interviews that many of them – at least half -  held a completely false belief about my client’s product.

During the interviews with these participants, they kept repeating the same idea.  The details are confidential to that client, of course, but the important point is that this audience really believed that their perception of the products was true. This wasn’t a minor technical detail – it was a major assumption about environments the products operated in.  But the problem was IT WASN’T TRUE! This was verified by my client, by independent evaluations that we found while investigating the conflict, and by some of the other participants in the study.

So here you have a very well established fact about a product, but at least half of the customer population believes something else. In other words: even if something is not logical, and even if your customers and prospects are technology vendors who tend to be logical, they may still believe in something wrong. And it could negatively affect their buying decisions.

This type of dynamic is extremely important for vendors to understand. If there are things that your audience believes about your solution that are not true, you’d better know about it so you can take action.

You’ll notice that I’m referring to this as a “belief”. Beliefs are harder to respond to than competitive FUD or outright lies.  These are things that your target market thinks are true because they’ve never really thought about it.

A formal market research methodology is the best way to uncover and to root out these false beliefs across your target market. You can’t rely on day-to-day conversations with customers, because if these issues do surface during such a conversation, you tend to correct them immediately with that specific customer and then forget about them since it didn’t make any sense to you to think that way.

In my last post, I talked about the similarities across Corporate IT groups when it comes to pain and product benefits.  This week I’d like to talk about key differences that are important to consider when doing research.

This is where segmentation comes into play – making sure you have enough participants of a particular type to represent the differences in your audience. How you segment, and whether you segment at all, depends on your goals. 

For example, if you’re doing a message validation where the goal is to evaluate if you are communicating well about a specific pain or the benefits of your technology, there are a lot of similarities between different geographies and verticals. In this scenario, you don’t need to have a dozen participants from each of the 10 vertical industries you sell into to quickly get a very real sense of how your messages are resonating with a corporate IT audience.

However, in some scenarios, you will need to get representation from different areas. Of course every project is different, and we at Dimensional Research work with our clients to really understand specific goals and make sure we recruit the right representation from any segment to reach those goals. Here are a few general areas to consider. 

The “Silos” of IT

Application Development, Production Operations, Security, …  They all act very differently from each other. They have different cultures:  development likes new features, while “change” often equates to downtime for production operations, and security would ideally prefer that nobody gets access to anything.  They purchase differently.  And (let’s be honest) these frequently have challenges communicating with each other.  If you have a product that crosses IT silos, you need representation from each silo to understand purchasing behaviors, objections, etc. 

Technical Buyer vs. Economic Buyer

The technical buyer will discuss features and functions.  This is a comfort area for many technology companies and they’re more likely to “get” you and your product.  However, the economic buyer is vital to tell you the non-product reasons why people buy – brand, sales execution, channel, financing, etc.  You will need both types of participants to understand competitive dynamics and purchasing behaviors.

Verticals

Government, Education and Non-profits can purchase very differently than IT in other vertical industries.  If you are targeting those verticals, you need to talk to them specifically about how they buy. 

Financial Services spends more on IT and is most likely to consider a new technology for competitive advantage.

ISVs and SAAS are organized and make purchase decisions much differently than corporate IT – although they often are an important market for technology vendors.

Geographies

There are big differences in how these geographies actually purchase products.  Culture can play a big part in acquiring a technology.  European countries frequently prefer working with local representatives – regional offices or channel partners.  Japan absolutely requires local language and culture in their purchasing process.  Canada acts like an extension of the US in the way they purchase and can be easily combined with a general North American study.

Technology and Process Maturity

If you are selling a cutting-edge technology, the best places to do market research are the San Francisco Bay Area, New York, London, and Sydney.  These are all cities with cultures of trying new things. 

On the other hand, if you want ITIL process maturity, Europe is great. And I absolutely love the mid-west (Chicago, Minneapolis, etc.) for identifying “objections” in a very straightforward and honest way 

Bottom line: You should work with your market researcher to figure out your goals and to determine which segments of the market you need to talk to. Then make sure you get the representation from these segments when executing your market research project.

I worked in vendor marketing roles (product marketing, product management, campaigns, etc.) for the first half of my career, before becoming a market researcher.  When I started doing technology market research and spending my days talking to corporate IT without any sales motivations, the conversations changed and I learned a few things that really surprised me and I wanted to share them with you. 

1. Customers do not spend anywhere near as much time thinking about your product as you do. Even customers that use the product on a daily basis spend most of their time thinking about other things – from people issues to process issues to the other 14 tools they use every day.  They really appreciate it when you make your product and messages simple so they can get on to the other parts of their job.
2. Customers really WANT to give feedback – happy or cranky, mature or brand new.  They want to talk about their experiences.  Some are very busy and don’t have time, but in general customers love being asked to join customer advisory boards or participate in research projects. There are a few caveats of course:
   • They don’t want to jump through a bunch of hoops to get permission – but that’s easy to solve by offering them anonymity for participation.
   • They want their feedback to go somewhere and yield real results.
   • They want to talk about the products they own not just the new stuff you want to sell them.
3. Customers HATE IT when you change the names of the products, or product features, they know.  It took them a lot of effort to learn your product, and they don’t like having to relearn everything, or struggle to talk about a product they love because they don’t know the lingo anymore.  And just to be clear, this is not a “minor annoyance.” It’s a HUGE negative.
4. Corporate IT is surprisingly consistent across companies and markets. There are differences of course, but there are also a LOT of similarities.   <UPDATE:  THIS POINT GENERATED A COUPLE OF QUESTIONS ABOUT WHAT IS DIFFERENT ABOUT CORPORATE IT, INSPIRING THIS POST.>
   • Geography: America, Europe, APAC all talk about the same pain, and same benefits of products. There are definitely differences in the way they buy products (channel vs. direct especially) and each region has different competitive dynamics and maturity of processes and technology, but for message testing or evaluating opportunities you don’t need to go far to get a view of the pains experiences by IT around the globe.
   • Vertical Industry: If an application is mission-critical, it makes very little difference if it’s financial services or manufacturing. The needs are very similar. Government, education, and non-profit organizations buy differently, but again, the pain is the same.

The discovery that surprised me the most? Realizing that customers actually WANT to give feedback and are willing to take the time to do it. As long as they know their feedback is impacting the company and its products, customers are more than willing to share it.

When you don’t listen to your customers or prospects, you risk losing them.

According to a recent report, “Did not listen to me” is the most widely experienced problem faced by 38% of services buyers. Additionally, 55% of buyers said they would be “much more likely” to consider hiring the provider “if they listened better.”  As Drew McLellan points out, if you add up the “didn’t listen” and the “talked too much,” that’s 63%.  Two thirds of your sales calls are being wasted simply because you talk too much and don’t listen enough! 

Market research is all about listening. It is also about asking the right kind of questions so that you’ll get the right kind of feedback. 

Here are three tried and true tips that I always use in my market research to make sure I actually listen to participants. These tips are applicable to every area of customer and prospect interaction.

1. Shoot for the 80/20 rule.  Always ask yourself, what percentage of the time are you talking, and what percentage of the time are the participants talking?  I shoot for the 80/20 rule.  If I’m doing more than 20% of the talking, then I’m probably not really listening.
2. Ask questions that encourage talking – not just “yes/no”.  Are you telling the person what their pain is, or are you letting them tell you about their pain?  A good researcher doesn’t say, “here is your pain – isn’t that right?”  They ask about pain to see if the pain they believe the prospect has actually comes up as a priority.  Of course you can always guide a discussion to keep it relevant, but you need to ask open questions in order to get real feedback – and give customers and prospects a real chance to say “no, I don’t care about that”.
3. Look for what you don’t know.  Never ask a participant “Do you agree?” 90% of the time they’ll say yes, because that’s the easy and polite thing to do. Your goal is to create an opportunity to hear something you DON’T know. 

To quote the Greek philosopher Epictetus, “we were born with two ears and one mouth, so we can listen twice as much as we speak.”  Thanks for the inspiration Drew.

I’ve sat through many advisory board and sales meetings with customers who have just been wined and dined. Then, during the formal part of the meeting, a product manager presents a new roadmap presentation and wraps up by asking, “what do you think?” with a face that clearly shows how proud they are of their work. 

You are very, very fortunate if you have customers that will actually give real, honest feedback in a scenario like that. They want to keep things positive and upbeat – they just saw pictures of that product manager’s kids or heard about their latest adventure trip. They don’t want to get all negative, even if that is what they really think.

One of the things that market research does really well is to intentionally CREATE an environment for honest feedback and for uncovering information you haven’t anticipated.  To create such an environment, you need to:

            1) Make sure the conversation is not about the personal relationship. Of course you want the relationship, and you want to leverage that. But the feedback can’t be influenced by whether someone likes you or not.  Of course they like you – you’re a tremendously likable person!  But the feedback must be about the topic, the market, and the pain points, not about what they think about you. 

You need to use verbal and non-verbal cues to set up the discussion so that it is open, non-judgmental, and the client is never aware that what they’re saying is bad news for you. You need to get the bad news now – not after you’ve been through a long, involved sales cycle that ends because there is “no budget,” which really means “not enough pain.”

            2) Dig for negative feedback.  Don’t just ask, “what do you think?”  Encourage people to talk about what they DON’T like.  You can’t make it better if you don’t know what’s wrong.  Ask, “What do you LIKE about what I just showed you?” then ask “What DON’T you like about what I showed you?” 

            3) Put feedback in context.  It’s not enough for a user to connect with the pain you solve – it has to be a priority!  I recently was in a focus group where a participant described a situation troubleshooting a particularly elusive problem in a complex application. It was a dramatic story with dozens of IT staff from different technology groups getting on bridge calls, pointing fingers, escalating issues, and finally getting the CIO involved in a 48-hour saga that ended up to be a rather minor configuration issue. I could feel the client behind the glass get excited because this was exactly the type of problem they envisioned their solution solving.

But then we asked how important it was to solve this kind of issue so it didn’t happen again. The participant explained that it wasn’t important at all.  It was the first time in 15 years it had happened and they had solved it.  They had moved on.  The pain wasn’t a priority.  Fortunately that group did express other pains that the client could solve, and they learned important lessons on ways of positioning their product that matched the most important pains.

Employing these techniques a great way to encourage your customers to give you real, honest feedback and to help you uncover pains you were unaware of.

Most businesses have an existing product or market that they are targeting, and their goal is to learn how to serve that marketplace more effectively. Any reasonable researcher should know that if you have a virtualization management solution, it isn’t that useful to talk about HR problems.  But while you need to stay on topic, you also don’t want to miss the chance to uncover information you are unaware of and may be a huge opportunity or a weakness you must address. This is one of the real advantages of qualitative market research where you can spend time deeply understanding the nuances of customer input.

Treat “unprompted” feedback with special importance.

Sometimes the information you don’t know that you need to ask about, is the most important information to hear.  Of course research must have a goal – but work with your researcher so they know your business and can use their judgment to understand what threads are worth digging into and what will take them into a rat hole that does not forward your business. 

The most important tool for unprompted feedback is this question:  “Is there anything else?”  This is the final question Dimensional Research uses for any market research interaction. After you’ve asked all your carefully crafted and specific questions, closing with an open-ended question can yield incredibly useful information.

Of course, this is a very important balancing act and research projects need to focus on their goals – not interesting side topics.  The core questions must be asked. But Dimensional Research makes a point of going through all input from every project with a special eye for the unprompted information.  Most of it is one-offs, with each participant having a special thing that only they are interested in.  But often you can spot trends in unprompted information that are really important. 

We’ve done projects where it turns out there was a critical development tool that needed to be integrated with the client’s product – not something they had realized was vital until the majority of participants talked about it, unprompted, in a series of customer and prospect focus groups.

In another project we didn’t realize that customer loyalty was almost non-existent, until we went through the unprompted feedback from interviews with the highly satisfied, happiest customers, and discovered they regularly evaluated the competition – not a question that we asked about specifically.

So do stay on topic.  But keep your ears open for the “+1″ opportunities or customer issues you hadn’t recognized – especially if you hear them from multiple sources.  Possibly minor product changes or messaging tweaks can help you address a different pain, target a new and lucrative market, or address important weaknesses.

One of my favorite market research projects is message validation. When we go and get direct feedback on high-level product messages, I always know we are going to learn something very important that will dramatically increase the results of marketing programs.  Hard ROI is easy to demonstrate when your marketing programs get twice (or more) the response from the same spend, because your message is more compelling!

So how does message validation work?  We evaluate your messages – the benefits and features that you give your PR department, your marketing programs team and your agencies – and break them into digestible chunks.  We set up one-on-one interviews with both customers and prospects (messages are experienced in isolation so this is not appropriate for focus groups).  And we walk through a systematic process of vetting the messages. 

We never ask, “What do you think?”  That doesn’t force participants to share what is really going through their heads.  You need to go deeper.  Instead of “what do you think?” you must ask, “what is MOST compelling?” and “what is LEAST compelling?”.  Do that for each area of messaging that you have: pain, benefits, features, positioning paragraph, etc.  Specifically prompt for good and bad  – and always ask WHY.  The “why” helps you know if you need to throw out some messages, use different words to express the same idea, re-order the importance, or segment your messaging.

What is interesting when you do this is how small changes can make a huge, measurable difference to your programs.  On the first message validation project I did in my career, we tested four main messages that the marketing team had been using for outbound emails and direct marketing with uninspiring results. The client knew that if they could get that 1-hour meeting, the clients were hooked, but the programs just weren’t getting the same level of response.

We tested the messages as part of an overall research project, and found that 3 out of the 4 messages resonated extremely well with everybody.  But that fourth message was different. Two out of our three target segments not only didn’t find it compelling, they actually found it insulting and frustrating. 

That said, it wasn’t that the fourth message was bad across the board.  The final target segment really liked the fourth message, even though it was actually alienating the majority of the market!  The client reserved that fourth message for a campaign targeted to that specific audience.

Solid, compelling messages about your product are the basis for a successful marketing program. Message validation helps you fine-tune your messaging to ensure maximum effectiveness of your communications with customers and prospects – a great cost saver.  

Bonus Tip:  If you’re looking for budget to fund a market research project, consider adding a message validation component and using marketing program budget!

There is usually some form of compensation for participating in a market research project. For focus groups and in-depth interviews, research participants are usually offered some kind of stipend in appreciation of their time.  The question I get asked frequently is – how much?

I’ll start by saying that the stipend is usually NOT the primary motivation for participating in technology market research.  I find people are genuinely interested in expressing their opinion. They like to be heard; they like to hear about new ideas coming down the pipe; they like thinking they are influencing the market or the product; and – in the case of focus groups and customer advisory boards – they like to hear what is happening with their peers.  And let’s face it, corporate IT employees are typically paid reasonably well, so a hundred bucks isn’t going to really compensate them for trucking across town to a focus group facility, spending 2 hours talking to you, then trucking back home.

That said, the stipend is key in attracting the right audience, and is perceived as an important added benefit. So back to the question of how much.  The answer depends on a variety of factors:

1. What are you asking people to do?  Focus groups typically take about 2 hours of participants’ time. In-depth interviews usually take from thirty minutes to one hour. In addition, for in-person focus groups, the participants need to get to a specific location.  As a result, focus group participants are usually compensated more generously.

2. How big is the target pool of participants?  The harder they are to find, the more you should sweeten the pot to simplify the recruiting.  If you have a straightforward recruit like “application developers” or “network administrators”, there are plenty of those and you don’t need to have a particularly large stipend to entice them to join. 

However, if you need something really specific like “DBAs responsible for MATISSE databases”, or customers of a competitor that only has a few hundred users worldwide, you want to have a large stipend to make sure that you’re doing everything possible to attract the few people out there that match the recruiting profile.

3. What level of participants are you looking for?  CIOs and other IT executives make more money than sys admins or developers, so the stipends need to be higher to match their expectations for the value of their time.

4. Customers or prospects?  Depending on the context, offering a customer money to give you feedback may be tacky.  Particularly if it’s a customer advisory board meeting, the real value is the chance to be heard.  In this case a nice gift with the corporate logo on it is much more appropriate.  

5.  Give the philathropic option.  More and more corporations have implemented strict policies that employees CANNOT receive any gifts of any kind.  Dimensional Research always offers our research participants the option of donating their stipends to the charity of their choice. 

6.  Web surveys are different.  When you’re looking for hundreds or even thousands of respondents, any stipend, no matter how small, can quickly blow up your research budget. There are certainly options like being entered into a drawing for a gift certificate.  Or the ever popular drawing for the hot electronic item of the day – the iPod Touch has been particularly common in survey drawings in the past year. 

One thing to consider, if the findings are not confidential, is to simply offer a copy of the final report to participants.  This is a very high-value offer to people who care about the topic – probably even more than a gadget – and has the added benefit of not attracting participants who don’t care about the topic and might not give the most insightful or informed answers.

There are many ways to make sure your online survey is efficient and effective. One of them, of course, is to avoid asking bad questions.  

Tip 1: Craft your questions carefully to avoid unwanted results 

This tip was inspire d by Seth Godin. He explains that “Every question you ask changes the way your users think. If you ask, ‘which did you hate more…’ then you’ve planted a seed.” 

Mr. Godin makes a great point. I recently booked a trip with Travelocity. It was a great trip. I was happy.  I did have a small issue with the airport transfer getting home, so I filed a complaint to see if I could get a refund.  It was a small matter – about $30 credit - so I wasn’t too worried about it. Travelocity’s reply email was pretty typical, asking me for more information (which I had to get by opening up the email THEY sent me, so a bit annoying). 

BUT… Then they sent me a survey asking about my experience with Travelocity.  One of the questions on the survey was “Do you know about Travelocity’s guarantee that your booking will be right, or we’ll work with our partners to make it right, right away?” I actually didn’t know, but this question clearly did NOT describe the experience I just had. 

Travelocity did make it right – it took them about 30 days to do so – but that wasn’t what their guarantee said and their survey pointed that out to me.  By including that question in the survey, they planted a seed that they didn’t want to plant and I ended up being less happy overall than I was before the survey.

Tip 2: Ask at least one question that participants actually WANT to answer

It’s important to ask the questions your customers are actually interested in answering.  Too often, marketing departments are so focused on the company’s newest offerings that they ignore the products their customers have come to depend on.

I use QuickBooks for my business. I have LOTS of feedback for Intuit on the core QuickBooks product, but they never ask me about that.  They constantly survey me, asking if I want to buy checks or do payroll or take credit cards, but there is no “Thanks for your time, is there anything else you’d like to tell us?” that would enable me to give them the feedback I WANT to give them.  

Another example from my own professional life: I use Zoomerang for my surveys. It is a GREAT product – with a few caveats.  One of the problems I have with the product is that they have a horrible interface for “choose the answer that most closely applies.”  It’s a small button that’s almost impossible to see. On the other hand, their interface for “choose all that apply” is great – a nice square with a big check mark.  I want to tell them about this issue, but they keep sending me web surveys about other things and I’ve never had a chance to give them important feedback that I really want to give them. Maybe they’ll read my blog and I’ll get to them that way?  <CORRECTION:  They did in fact just send me a survey this week that allowed me to give that feedback.  I’ll wait and see if they act on the feedback.>

Tip 3: Reward your participants, wisely

This tip is inspired by Patricio Robles: “offer users who respond to a survey a discount, an entry in a drawing for a prize, something of value. It will boost response rates and make them feel like they’re investing their time wisely.”

While I generally agree, I would add that when offering a reward to participants, it’s important to consider your target market. A gift that is too nice motivates people who aren’t qualified to complete the survey. Then you have to wade through junk or set up lots of qualifying questions to weed them out. If the reward is nice enough, some people will game the system and try to guess what you’re looking for, so the filtering questions won’t always work. 

When you’re doing a technology web survey, this is especially important, since people who don’t know your topic can really throw off the results, as they are not informed.  If possible, giving a copy of the final report on a topic is a great incentive, since only people who know and care about the topic will respond to this type of reward, keeping your input very clean.

Dimensional Research clearly stands apart from other market research firms because we are technology experts, performing market research for technology companies.

But what exactly does it mean to be a “technology expert” when it comes to research? Obviously it doesn’t mean we’re deep technical experts in every possible area of corporate IT. We don’t write code or manage developers. We don’t manage data centers (although we do our own IT and can commiserate with some of the pain!). We don’t personally install patches or upgrades or test new releases of enterprise applications.

What we’re experts on is Corporate IT and how it works. We are very familiar with Corporate IT and its processes and can talk comfortably and knowledgeably about:

1. Technology. Not the actual coding and implementation, but the types of technology and the tradeoffs.  We understand the perceived benefits and potential downsides of a service-oriented architecture.  We know what cloud computing is (or at least, know all the ways the industry uses the word) and what is real and what is vision.  We understand the challenges with data and the effort that is needed to generate simple graphs and reports.  We know the TLAs (Three letter acronyms) – from IDE to BSM to SDL to SAN (Both the storage one and the security one!)

2. Processes. We know that ISVs and SAAS companies purchase differently than financial service organizations and that those are different than government and non-profits.  We know the software development lifecycle, the people involved, and the products used. We are familiar with ITIL and know how some organizations treat it as a guiding principle, others as one of many guidelines, and some just ignore it.  We know the new focus on costs and services in ITIL v3.  We also know IT has other systems – Six Sigma, COBIT, and more that need to be considered

3. People. We know the common organizational interactions that take place between IT and the business owners; between applications and operations groups; between development and testing; and between server and network teams.  We love talking to engineers and technologists about their challenges, and appreciate the creative side of a job that is often considered to be purely technical.

Most importantly, because we have this foundation, we can quickly come up to speed on any new technology. 

• On a recent security project we had to learn new vocabulary like “pen testing” and “cross-site scripting”. It didn’t take us long, because we already knew the development and testing processes and quickly understood where security fit into those.
• On our first data center automation project, we had to learn about different types of automation.  But the products automated processes we were already familiar with, so it was a very straightforward learning curve. 
• When we did our first cloud computing project, our foundational understanding of virtualization, systems management, and application management meant we were able to be productive right away.

Being technology experts means we “get it.” Whatever technology our clients are developing, whatever processes they use, we are very comfortable talking with IT using professional terminology, asking the right follow-on questions, and gaining the respect of the participants. The result: an efficient and focused market research process that generates a highly effective analysis.