Are customer surveys worth the paper they are written on?

Testing has become a part of our culture, but it didn’t start out that way.  I remember back in 2009 when we first began using Google Optimiser we were making subtle changes to our printer ink website and tracking them with Optimiser. We were seeing a 40 - 50% increase in conversions from the changes in the first couple of days, and I had dreams of retiring to a beach in the Bahamas. However after about a month of running the test, when the numbers became statistically significant we actually saw very little difference and I was brought back to reality!

This is my point in this blog; you need to understand what the results of testing or customer feedback are actually showing you before you go all out and implement changes. In the case of Optimiser we now know that subtle changes are too small to be recorded accurately; you need to be testing big changes. Also you can’t take the first few days’ results as Gospel, you have to give it a month or more before drawing any conclusions.

Likewise with customer feedback. We have always had some means of garnering customers’ views on the website, and my philosophy has been to use the positive comments to develop and push our unique selling points (USPs), and to use any negativity to address problem areas as a matter of urgency. However, when Walmart got itself into a customer-driven change mentality it cost them an alleged $1.85 billion - not an insignificant amount I’m sure you will agree.  

According to a story on Daily Artifacts, back in 2008 Walmart conducted a customer survey which found that the majority of customers wanted a “less cluttered” store, so it rolled out a strategy which revised its decades-old strategy of low price and wide selection.

The revamp not only removed items but cost "millions of dollars" per store in refurbishment costs. The exercise saw an immediate loss in revenue and decline in same-store sales data (see the chart by clicking on the image at the top of this post).

In hindsight Walmart relied on what customers said in a survey, rather than what it already knew from customer behaviour in store. It took this research as the truth and rolled out a huge programme of change that affected the value of the store group to the tune of an estimated 1.85 billion dollars. Some argue that this happened at the same time as the global recession was hitting people’s spending power, however Walmart with its low prices and value model, should have been in a prime position to benefit when budgets get tight.

So, the rather expensive lesson that we can all learn from this is:

Always listen to your customers, but sometimes, just occasionally, they might be wrong!