Market Research disrupted

This article was published in the September 2016 Issue of the Online MR Magazine – CLICK HERE to read the magazine.

emile-bakkerSweet were the days when all information used in business decisions was provided by Market researchers. Just as sweet as the days that research customers were interested in advice, not the data. These days are long gone. And as it always is with memories… they always get sweeter as time passes by.

What happened since then? Changes are a chain reaction so I’d like to look at what is changing at the customers of Market Research that is forcing Market research to change.

There are two main ingredients as a source of change: Data abundance/availability and process agility.

In-company data generation

Triggered by technology advancements that resulted in CRM systems, helpdesk systems, IoT, e-commerce, etc,  the amount of data a company creates, uses, collects and needs to manage has exploded. During the past decades businesses learned to manage this data and to utilize it. Data is not only part of a process (like a helpdesk ticket) but it is also used to measure. All data coming from processes in organisations is used for real-time monitoring and evaluation as well. These developments are unstoppable. It started by using helpdesk ticket data for helpdesk employee evaluation and has moved to applications like “Connected Tyres” of Michelin. And those do exactly what you expect with such a product name, each tyre has sensors connected to the internet. This last example allows monitoring the tyres of individual vehicle, but also serves as a source of data for future developments. The data serves an operational goal as well as a research goal.

Agility of process and product

As the volume and variety of data in companies started to grow, companies discovered that this, often operational data, could be used to evaluate service effectivity and validity of product specifications. This resulted in a massive change of business processes. These processes changed from “focus on sticking to the plan” towards “continuously learn and adjust to achieve goals”. In other words, data generated in an organisation became part of a feedback loop used for optimisation. Change and tune became the goal, and static focus became a bad thing. The best example of this is how e-Commerce companies test website changes. They make the change, perform an a/b test on life traffic, and let the conversion determine if the change will be rolled out across all users.  Agile, scrum and lean are the modern project management methodologies based on these principles.

What does this mean for Market Research

So what are the implications of these two elements on Market Research. In short the first conclusion is that over the last two decades the Market Researcher has lost it position of the sole provider of insights.

A second conclusion is that the moment of need for an insight is far more distributed across the business processes. Since the increased agility of companies allows them to tune at any moment, the data used for that tuning is actually coming out of processes and not necessarily research. A simple example is the page optimization in the eCommerce world.

So now we have a situation where the data coming from Market Research can be seen as additional data points, yet, the need for insight has become a continuously need across all available data.

How does this change the added value of the Market Researcher? A researcher can choose two directions; Go in depth or go wide.

Market research in depth

This is the researcher that accepts that what they deliver are “additional” data points. However, in order to deliver value with this, these data points have to be extremely valuable. So in order to deliver this value the researcher has to build up an extreme knowledge and translate this to specific models, methodologies and technologies. Think of specific services like customer feedback loops or continuously competitor monitoring and analysis. Also complex qual research can be seen as the in depth choice. Most of the in depth services are continuous and directly connected to data sources of the customer.  The outcome of these projects and systems are mixed by the customer in their data soup and they, independently from the researcher, come create their Insights.

Go wide

This choice will has as goal to be the provider of total insight across entire organisations, products, or markets and data. This means that besides research knowledge, specific industry knowledge is needed. Researchers that like to go this way will have to embed themselves in the data streams of their customers, integrate additional data points, set up models, dashboards and analysis in order to provide their services.  This service comes close to the traditional management consultancy services delivered, but far more data oriented.

New skills needed

So over the past two decades the required skillset has changed. While in the past the focus was on the efficiency of a research process, it now has moved to being able to use and embed data generated by the research customer and third parties. Whether you like to go in depth or wide, you will be confronted with a research process that uses data coming from different sources (frequently even in real time). This makes, literally, every project different and a technical data challenge on its own. Utilizing technologies to connect to data sources that deliver data in batches or (transaction) message based, validate and process incoming data, enrich data across sources, and generate real-time insights are common needs. As an example think of feeding a real-time research model based upon website transactions.  This can only be part of your service if you master data and technology.

Another important skill needed is a deep understanding of the processes of your customer. Since the focus of organisations is on tuning, the market researcher needs to be able to provide embedded services in the customer processes. The researcher does not deliver the input of a process, but is part of the process. So, any researcher needs to understand project management principles used by the customer in order to add value. And exactly these have been heavily innovated over the past 2 decades. Methods like scrum, agile and lean have become the way marketers work.


Whether you go in depth or go wide, modern research is all about embedding. In process and technology. Data skills that goes beyond pressing a button in SPSS are needed, terms like ETL, Hadoop, Hive, R, streaming, events, etc should be known by you.

It is all about your understanding of, and embedding your service in your customer’s business process.

The good thing of these changes is that people now expect Insights to be real-time and continuously, so projects that run for years are no exception. And that justifies a higher investment in technology needed in these projects.

This article was published in the September 2016 Issue of the Online MR Magazine – CLICK HERE to read the magazine.