Archive for September, 2011

Questionable questionnaires ?

Saturday, September 24th, 2011

We see them here. We see them there. Questionnaires now seem to be everywhere.

Everyone appears to be asking more and more questions.

This is generally good, right? Or is it?

My philosophy has always been that any research is better than none but now I am not so sure.

Doing some research poorly is probably worse than doing none at all, in fact. I am reminded of a quote from Ruth Mc Neil, a fellow of the Market Research Society, who responded to the Sony Walkman research debacle – when research results suggested that the product would bomb with consumers – by saying “People who use the wrong research get the wrong result. If the product had been described properly, consumers would have realised that they did need it after all”.

This thinking can be applied to the boom in Do It Yourself (DIY) surveys. If you ask the wrong questions, in the wrong way – you will get the wrong answers! This could lead to making unjustified and potentially costly decisions.

For many years now, I have been on a quest to find the perfect questionnaire. I have yet to find such a thing or, as yet, to create one myself! I am not convinced that such a thing exists!

There appears to be a similarity between people’s views on questionnaire design and driving skills. Everyone appears to believe that they are able to do it and can do it very well! As we know, the difference between this perception and reality can be very different!

Like any other skill, designing effective questionnaires and selecting the right methodology for ensuring that the right people are targeted and that response rates are maximised through the delivery mechanism and so on needs to be learnt and developed. We are not born with an innate ability to design exceptional questionnaires!

Online DIY survey software enables anyone to get in ‘a car and drive’ (design a questionnaire) sometimes with disastrous results.

So where am I going with all this? I went for a pub lunch with my family in Essex. The pub shall remain nameless.

Comments Card

At the end of a very good lunch, we asked for the bill and, as is often the case now, a ‘Have Your Say’ customer comments card came with it! At face value, I thought this looked good and saved it for use in future marketing college delegate workshops!

On closer inspection, I noticed some issues with it.

Introduction

  • Very good to see that there was one. The wording was a little odd though – “To help make your experience as enjoyable as possible we’re asking for your opinions” – Sounds OK except that this is being asked at the end of the meal so the respondent is about to leave.  
  • Classification questions right at the top – Name and Email address. This takes up a quarter of the space available which is perhaps a little much. It does say that “Any information you provide us with is strictly confidential and will not be passed on to any third party”. Fine,  but presumably they intend to use it for their own direct marketing purposes but they do not say so.

Ratings

  •  They made good use of rating scales although I am always slightly concerned  by  the use of “Okay” as ‘middle ground’.

Open questions

There were two of these.

  1. “Please comment on your experience and suggestions on any improvements”

 

It was going so well! Here we have a double barrelled and ambiguous open question. Unfortunately, it is badly written so it doesn’t make sense either.

  1. What would you like to see changed or improved here? (E.g. theme nights, music, certain dishes etc)”

Sorry to be pedantic but changed or improved would be better as two questions perhaps. Are the examples given changes or improvements to the meal and or the service which is what they have been questioning up to this point? It is unclear.

Multi choice

“What age group are you in?”

This is fine as a classification question.

“Who was with you?”

Again, slightly confusing. One of the options was “Own”. Presumably, on my “own”? Or “no-one” perhaps?

“Are you doing anything else in the area” Options were “Work, Shopping, Leisure, Other (no space to add any details) or No”

What is the purpose of this question? It is unclear as to how they might use this to make ‘your experience as enjoyable as possible’.

 So, as I stated earlier, the comments card looked reasonable at first glance but this short questionnaire is trying to do too much. It is attempting to get insight on at least seven areas

i)                    Volunteered data so that they can market to you

  1. Age
  2. Name
  3. Email address

ii)                   A rating of the meal

iii)                 A rating of the service delivery

iv)                 Comments on the experience?

v)                  Suggestions for improvements  to the experience? (assumed)

vi)                 The type of customers that are eating in the pub?

vii)               What changes or improvements are desired for the pub as a whole? (assumed)

This is too much for one side of a card 21 cm by 14.5 cms!

Of perhaps more concern is that they are missing the opportunity to get more  depth of response on key elements such as the venue, location, facilities e.g. parking, atmosphere, likelihood of returning etc…

This may appear very critical but I regularly see much worse.

The message?

 Research needs to conducted using robust methodologies that deliver against clearly identified objectives.

 By all means create your own questionnaires or surveys but perhaps consider taking some ‘driving lessons’ to hone your skills too!

 We offer a service that can help to improve your questionnaires called The Quest. Find out more about how you can get your existing or draft questionnaires vetted here.