Does your survey reflect reality, or is it just wishful thinking?: Lessons from the Mirror of Erised

Old Mirror

Old Mirror, by Sthetic, on Flickr

One of my favourite devices in the first Harry Potter book, Harry Potter and the Philosopher’s Stone, is the Mirror of Erised.   Harry comes upon this old mirror one day, and when he looks into its depths, he sees his (dead) parents standing behind him.   His friend Ron sees himself winning at Quidditch.  The mirror, it is explained, distorts: it shows the viewer their heart’s desire rather than ordinary reality.   (Incidentally, Harry nearly wastes away gazing into this mirror, hungry for the vision it shows him. But that’s by the by).

Only the perfectly happy person would look into the Mirror of Erised and simply see themselves.

Wistful dreaming of a perfect world is quite a feature of the surveys I get to fill in.   All too often, the people who create questionnaires and research studies seem to concentrate on all the things that are important to them, and utterly neglect the wider context.  The result is a weird distortion of the customer’s reality.

Facebook’s ‘Like’ button is a nice example of this.  ‘Like’ has become a vague indicator of the number of fans that a person or item has. It’s very little use an actual indicator of liking, because there’s no context.   To get some sense of real levels of liking, you might wish to know about dislike, or degree of liking.

In other surveys, I can complete page after page of grids, and yet at the end feel that the questionaire never really captured my opinion.    Perhaps they grilled me on my attitudes to, say, Bluetooth, without asking me if I knew what it meant, or whether I had it, or whether I was convinced it would fry my brain.

Of course, we’re all guilty of distorting our vision to a certain extent.  The problem is that the distortion can become so great that you can fail to collect the simple, essential information that will allow you to make sense of what’s going on.

Some tips on avoiding the ‘Like’ trap:

1. Always get the context

Spend time asking the basic questions.  The most important questions that I ask in an interview or focus group are not the ones in my brief: they’re the simple ones about who you are, what you do for a living, how you keep in contact with your friends.  This bedrock information will help you make sense of feedback.

2. Get an outsider’s view

Check your questions with an actual person who is not part of your marketing/PR/social media enclave, and then take up their suggestions.

3. If in doubt, just ask people what they think

You would be amazed how often no one thinks to do that.  An simple open question works wonders. More context, more emotion, more reality.

4. Accept the feedback

Lots of feedback is a bit random or even negative.  Some of it will be right outside your remit or indeed God’s.  Make sure there are ways of passing information onto the other people who can act on it, but then listen to what people are saying.

This can be damned hard, but sit there, quieten all the voices in your head which are yammering about the unfairness of it all, and just listen. What are people saying?

Now you’re in a position to make a meaningful decision.

POSTSCRIPT: I’m moving my blog over to a different host in the next few days. Please excuse any mess that may ensue.