Big Data Can’t Measure Instinct

March 19th, 2013

‘Big data’ is increasingly cited by commentators as the future of insight for brands and agencies alike, with 2013 already christened as ‘the year of big data’. In a study conducted by Winterberry Group which surveyed over 150 senior figures in advertising, marketing, publishing and technology, 77 percent said data management platforms will play either a “critical” or “major supporting” role in analysing and improving their advertising and marketing efforts in the long term.

This isn’t just about marketing either – brands are using consumer data to create and improve their services and products – the Nike + and Fuelband ecosystem being just one example. Netflix has even taken big data to the next level – using in-depth knowledge of their consumer’s viewing habits in their new venture as content producer with their new series House of Cards.

Despite the hype, this isn’t a new phenomenon: the use of algorithm and analysis to make sense of the huge streams of information left in our digital wake has been on the cards for some time. In 2008, Wired’s Chris Anderson wrote:

“This is a world where massive amounts of data and applied mathematics replace every other tool that might be brought to bear. Out with every theory of human behavior, from linguistics to sociology. Forget taxonomy, ontology, and psychology. Who knows why people do what they do? The point is they do it, and we can track and measure it with unprecedented fidelity. With enough data, the numbers speak for themselves.”

But this might be just the problem – the numbers can’t always speak for themselves. Or rather, they can say a number of things, depending on who’s reading them. What’s missing is a more human intuition about the consumer – vital to making sense of data sets, giving them a context beyond the numbers. And increasingly, companies are struggling to use the data to make the decisions that matter.

In his final all-staff email after being fired, Groupon CEO Andrew Mason wrote:

“If there’s one piece of wisdom that this simple pilgrim would like to impart upon you: have the courage to start with the customer. My biggest regrets are the moments that I let a lack of data override my intuition on what’s best for our customers.”

Reliance on data (whether you have too much or too little) risks being at the detriment of a real understanding of the customer and their needs and motivations. For strategists and planners both client side and in agencies, this intuition about the consumer comes from being immersed in their world – in all its contradictions and immeasurability. Multinational brands like P&G are beginning to tackle this by sending their product teams in-field for days or even weeks to better understand their consumers.

In an article for the Boston Globe, Samuel Arbesman refers to the ‘bias towards measurable information’:

“Throughout history, in one field after another, science has made huge progress in precisely the areas where we can measure things—and lagged where we can’t… The result, over time, has been that we know a lot about the things that are closer to our size, our altitude, our spot in the universe—and less about things that are hard to reach, hard to dig up, and hard to quantify. What we know has a bias, in other words, and is biased in favor of what we can measure.”

So while the never-ending streams of big data coming out of social media, online transactions and smartphone use might seem to reflect a huge amount of information about the consumer’s life, it doesn’t reflect everything. It might be possible to track how someone shops, or how they interact with their peers on Facebook, but it would be a mistake to imagine that this represents how they are all the time, in every situation. We also need to be aware of assuming that this data even represents all people – 76% of people globally still don’t use smartphones, and while it may seem that Facebook is everywhere, only 14% of the world’s population are active users. As Arbesman says, “big data might be deep… but it’s not wide”.

It’s here that human intuition about the consumer comes into play – data tells us that something happened, but not the complex emotions and drivers behind it, the ‘why’. A balance needs to be struck between using big data to see behaviour on a grand scale, while acknowledging its limitations. Offsetting huge data sets with a more intuitive understanding of the small-scale drivers and emotions behind behaviours is vital to avoid the perils of believing in numbers over people:

“Data-driven predictions can succeed — and they can fail. It is when we deny our role in the process that the odds of failure rise. Before we demand more of our data, we need to demand more of ourselves.” Nate Silver: The Signal and the Noise: Why So Many Predictions Fail — But Some Don’t.


Millie Findlay works at creative consultancy Sense Worldwide, where she works with brands like Converse, MTV and Pepsi to help them make things better and make better things. Millie is a regular contributor to Junior Strategy and you can read more of her here


Leave a comment: