Category — Web Surveys

KACE – the leading systems management appliance company and a fabulous client - just announced the results of a new Dimensional Research Web survey that revealed 84 percent of IT staff polled do not have plans to upgrade existing Windows desktop and laptop systems to Windows 7 in the next year—despite early enthusiasm from beta testers of the new operating system

Other key findings include:

• 84 percent of survey respondents have no plans to upgrade existing Windows desktops and laptops to Windows 7 next year
• 72 percent indicated they are more concerned about upgrading to Windows 7 than staying with an outdated XP operating system
• 50 percent revealed they have considered moving from Windows to an alternative operating system, and 27 percent of those cited Mac OS as the top alternative
• Almost 60 percent of survey respondents do not presently have a tool in place that automates operating system migration
• Economic factors, such as budget freezes and staff reductions, were cited as other reasons to not immediately adopt Windows 7

You can download a full copy of the report, as well as detailed information on the methodology used at http://www.kace.com/resources/Windows-7-Adoption-Survey.

 Here’s a few links to some news coverage of the survey:

• Computerworld, Gregg Keizer
• Fortune, Philip Elmer-DeWitt
• The Register, Gavin Clarke

There are many ways to make sure your online survey is efficient and effective. One of them, of course, is to avoid asking bad questions.  

Tip 1: Craft your questions carefully to avoid unwanted results 

This tip was inspire d by Seth Godin. He explains that “Every question you ask changes the way your users think. If you ask, ‘which did you hate more…’ then you’ve planted a seed.” 

Mr. Godin makes a great point. I recently booked a trip with Travelocity. It was a great trip. I was happy.  I did have a small issue with the airport transfer getting home, so I filed a complaint to see if I could get a refund.  It was a small matter – about $30 credit - so I wasn’t too worried about it. Travelocity’s reply email was pretty typical, asking me for more information (which I had to get by opening up the email THEY sent me, so a bit annoying). 

BUT… Then they sent me a survey asking about my experience with Travelocity.  One of the questions on the survey was “Do you know about Travelocity’s guarantee that your booking will be right, or we’ll work with our partners to make it right, right away?” I actually didn’t know, but this question clearly did NOT describe the experience I just had. 

Travelocity did make it right – it took them about 30 days to do so – but that wasn’t what their guarantee said and their survey pointed that out to me.  By including that question in the survey, they planted a seed that they didn’t want to plant and I ended up being less happy overall than I was before the survey.

Tip 2: Ask at least one question that participants actually WANT to answer

It’s important to ask the questions your customers are actually interested in answering.  Too often, marketing departments are so focused on the company’s newest offerings that they ignore the products their customers have come to depend on.

I use QuickBooks for my business. I have LOTS of feedback for Intuit on the core QuickBooks product, but they never ask me about that.  They constantly survey me, asking if I want to buy checks or do payroll or take credit cards, but there is no “Thanks for your time, is there anything else you’d like to tell us?” that would enable me to give them the feedback I WANT to give them.  

Another example from my own professional life: I use Zoomerang for my surveys. It is a GREAT product – with a few caveats.  One of the problems I have with the product is that they have a horrible interface for “choose the answer that most closely applies.”  It’s a small button that’s almost impossible to see. On the other hand, their interface for “choose all that apply” is great – a nice square with a big check mark.  I want to tell them about this issue, but they keep sending me web surveys about other things and I’ve never had a chance to give them important feedback that I really want to give them. Maybe they’ll read my blog and I’ll get to them that way?  <CORRECTION:  They did in fact just send me a survey this week that allowed me to give that feedback.  I’ll wait and see if they act on the feedback.>

Tip 3: Reward your participants, wisely

This tip is inspired by Patricio Robles: “offer users who respond to a survey a discount, an entry in a drawing for a prize, something of value. It will boost response rates and make them feel like they’re investing their time wisely.”

While I generally agree, I would add that when offering a reward to participants, it’s important to consider your target market. A gift that is too nice motivates people who aren’t qualified to complete the survey. Then you have to wade through junk or set up lots of qualifying questions to weed them out. If the reward is nice enough, some people will game the system and try to guess what you’re looking for, so the filtering questions won’t always work. 

When you’re doing a technology web survey, this is especially important, since people who don’t know your topic can really throw off the results, as they are not informed.  If possible, giving a copy of the final report on a topic is a great incentive, since only people who know and care about the topic will respond to this type of reward, keeping your input very clean.

This morning I was asked to do a survey on backups by a friend of a friend who is doing an MBA project.  An interesting topic and one that is close to my heart as a small business owner who has had two issues in the past four years that I only survived because I run regular backups.

One of the basic principals of building Web survey questions is that the options for answering have to cover ALL POSSIBLE responses – not just the responses you think are most likely. 

So with that background, here’s the bad survey question of the day:

Q:  How long can you survive without the data and applications on your computer?

• A few hours
• A day
• A few days
• More than a few days
• Forever

My answer to this question is:  I CAN’T SURVIVE!!! 

Seriously, in my business I need my data and applications every moment.  But I didn’t have an option to present that opinion, and I probably answered all the other questions in a different way because this is so critical to me – which the surveyers won’t be able to segment.

This question could have been easily fixed by asking “How else could someone answer this question?”  With the one exception that I had, it’s a pretty complete list.  So maybe they could have realised they were missing my answer if they had just taken a moment to think about completeness. 

In general, adding an “Other” category allows you to quickly learn that you have missed something – a great best practice because even the best researcher doesn’t know all the options.

Dimensional Research’s web surveys are one of our most requested services, for good reason. Many research companies often dismiss online surveys simply because they’re not as profitable. But while web surveys do have several drawbacks, they can be highly useful. As long as the disadvantages are understood, and Web-surveys do not become the only research tool in the toolbox, Web surveys can yield very powerful results.

So what’s WRONG with Web Surveys?

Web surveys are a quantitative research vehicle. As such, they are highly efficient, but easily miss contextual detail. They make it impossible to capture subtleties. When using online surveys, the researcher doesn’t get an opportunity to drill down into the research topics, to ask the participants clarifying questions or to uncover hidden issues.  The all important research tools of “why” and “tell me more about that” aren’t available.

A big challenge in crafting web surveys is question design. Designing the right questions is incredibly important because once the survey is fielded, the researcher doesn’t get the opportunity to discuss the questions further with the participants, to clarify how words are understood or to capture the subtleties of the answers.  Unlike qualitative research, the survey is only as good as the question-writers understanding of the problem.  There is very little opportunity to learn anything beyond the scope of your current knowledge.

Then, what’s GOOD?

Web surveys are an inexpensive, cost-effective way to get a broad range of input. They can be executed quickly to reach a large number of respondents in multiple geographies. Internet surveys enable researchers to collect a large amount of data in a relatively short amount of time, and to draw responses from all over the world, resulting in a dramatically larger sample size than they could get using qualitative research methods.

With today’s online survey tools – Dimensional Research uses Zoomerang and Survey Monkey –  reporting is immediate.

It is relatively easy to recruit participants for web surveys because they involve minimal time commitment. This is especially true for participants who are technology professionals, since these participants are very comfortable with online tools.

So, back to the original question – is a Web Survey right for me?

Maybe.  If your goal is any of the following, Web Surveys should definitely be considered:

1. Capturing the hard data you need in order to capture customer pain points, support your business plan, or prioritize product features
2. Documenting market conditions for use in outbound marketing efforts.
3. Gathering information quickly when time is the most pressing consideration.