In August 2011, Marc Andreessen wrote a prescient opinion piece called “Why Software Is Eating the World.” Andreessen pointed out that even in the depths of a global recession, a remarkable thing was happening as bold digital startups wrested control of industries like bookselling, moviemaking, music distribution, telecommunications, transportation, etc., from legacy competitors. We believe, a similar tsunami is about to hit market research, and will soon disrupt ‘market research’ as we know it – out of existence but interestingly into a 10X more powerful function!
The Market Research Shake-Up…. finally!
The fact is, market research today looks very much like it did 15 – or maybe even 30 years ago. The impact of technology on market research has been relatively shallow – with some automated processes and more frequent online data collection. AI in its various forms such as NLP (Natural Language Processing), Machine Learning & Neural Networks is increasingly being tried by various startups and companies. However, we have not seen anything that completely changes the nature of the market research process itself. Most of the applications today are about doing something incrementally better than yesterday.
Today market research still remains people and time intensive. There still is a significant time lag between data collection, insight generation and impacting business decisions. Insights are still shared through PowerPoint decks and may be some dashboards. Research data from a project is typically used to answer the business and research questions for a specific study only. Most of the critical information, skills and insights are with the research team – consider how much information is lost when an experienced research team member leaves the organization. These are just a few examples of how today’s market research fails to deliver on repeatability, reusability, and scalability – all very critical elements for today’s growth focused organizations.
Now imagine a different world where research projects are launched without any human intervention, autonomous adaptive surveys automatically modify, drop and add questions during data collection, insights are generated and shared within seconds of data coming in, research data is actively combined with data sources such as sales, behavioral, IOT and passive data to create recommendations for all strategic and day-to-day decisions, business users are proactively provided the insights as they make the decisions, anyone can ask a question in natural language and get the answer back same way.
Now wouldn’t that be something. Interestingly, all of this is possible today. It is not necessarily being done today but we are not far from each one of these fantasies being real.
We are betting on two big trends that will make this AI enabled world possible.
- Algorithms – not rented ‘skills’
- Impact – from X to ‘10X’
Algorithms – not rented ‘skills’
AI algorithms “today” can perform many of the functions currently done by experienced and skilled market researchers. These algorithms can create research designs using analytical frameworks and business context, craft questionnaires, conduct analyses and generate automated insights – that too, in the relevant ‘business’ language. While, all of these are in the early stages of development, given the speed of innovation, we expect AI algorithms to surpass the average market research skills within a year or so.
The big implication of these emerging AI algorithms is that companies will build or rent these algorithms and limit their dependence on vendors to conduct research. Once the algorithm is built, it’s easy to apply in new ways, and it only gets smarter with time. These algorithms will ensure that market research is scalable and thereby make it possible to deliver value across the organization. This also means, that the strategic advantage will be with those organizations that have superior skills to build and roll-out such algorithms. Even medium and small companies will have access to the superior research skills that today are typically available only to large enterprises.
Impact – from X to ‘10X’
Market research today is primarily used for driving big strategic decisions and for monitoring critical customer related KPIs. This means that there are a ton of day-to-day decisions that are not at all impacted by the research data. There are many reasons for this, but the primary one being that it is very difficult to use market research data in such decisions as the data is not at the right level of granularity, is typically not updated frequently enough and is difficult to provide to the decision makers in a format that they can use. With AI based decision recommendation platforms, we are already seeing examples of research data being embedded into day-to-day decisions at leading organizations.
One of the most dramatic changes of AI-driven market research is that unlike the traditional approach, the output from AI-driven market research can be quickly combined with data from multiple non-traditional sources — including sales, text, social media, behavioral, passive data etc. — and yield new recommendations that reflect a changing marketplace. These updated recommendations can be made available to the decision makers within a few minutes of data collection. The combination of non-traditional data sources with market research data is very powerful as this provides a more comprehensive and complete view of the market place – resulting in significantly better decision making which translates into a significant competitive advantage.
Further, while most of the data sources used in analytical applications such as sales data, behavioral data and passive data can provide up to minute information for decision making, they fail to account for any trends related to the future – this is one critical information that only market research data can provide. Hence, while AI without market research data can optimize a decision based on the latest – but past information, with market research data, it can optimize decisions for the future.
This fast and wide impact on business decisions is what is going to drive the impact of market research from X to 10X in the future.
The Future of Market Research
Market research used to be a strategic asset, focusing on “why” consumers made specific decisions. Now, it’s evolving into a strategic AND a tactical tool, answering not only “why” but “what” and “how” to drive decisions on a daily basis. In the future, market research will be more scalable, and projects won’t require as much human expertise to develop frameworks and add business context. In the coming years, market research teams will focus more on understanding how AI based technologies can bring in different data sources — observational, passive, video, social media, etc. — and use algorithms to generate not only insights but specific recommendations. Moreover, research will become faster, cheaper and more insightful as the researchers will focus on areas that AI cannot do – generating insights beyond what AI produces, storytelling and identifying areas where each research project can add additional value beyond the specific business question that drove the research.
Some people in the market research industry may not want to hear it, but change is coming, and it’s going to be big and moving really fast. The good news is that market research that is faster, cheaper and better will democratize the function, ensuring it is used more widely by more companies to drive more decisions. Market research professionals who embrace this new reality and adapt to it will be in a better position to help their clients develop and sell products- for market research, as we know it, will come to an end as AI-driven processes take center stage.
