How a Good Concept Test could go Bad

One week has passed since Alex decided to employ concept testing to determine if his product idea would be celebrated by the market or tossed in the bin marked ‘Failure’. Today, he approached his boss, Jane, and explained his work plan.

A big part of his plan was concept testing. He expected that the road ahead would be very smooth: they would get a research partner on board, test the concept, and find out if the idea would realistically work in the market – or not.

Jane, unfortunately, did not seem as excited about his brilliant solution. In the ten years she’d been in the industry, she’d seen several such studies unintentionally mislead product managers into getting rid of brilliant ideas and launching relatively weaker ones in their place. The core issue, as far as Jane was concerned, still remained: To launch or not to launch!

It’s not that Jane doesn’t believe in concept testing. It’s that she knows how easy it is to get important elements of it wrong, thus leading to inaccurate, easily-misinterpreted results.

After thinking for a moment, she said to Alex:

“I’m not against concept testing, but it’s important to do it right. Before you start testing, make sure you identify and use the appropriate KPIs and benchmarks for scoring. Let me know if you need help with that.”

How will Alex do this?

Choosing the Right Key Performance Indicators (KPIs):
Research agencies often focus on incorporating product-relevant KPIs in the research design, for example, test for likeability, relevance, uniqueness, etc. This allows the researcher to estimate how well the product will fit the market. However, many times our business objectives aren’t limited to simply launching a product. There are other overarching business goals that need to be considered. It is critical to reconcile the KPIs to both the product and to these broader business goals. For example, a concept might test as relevant and unique but fail on the business goal of getting 20% new people into the category.

If Alex can choose KPIs that speak to both the product and the big picture, this will give him accurate success indicators. This, in turn, will make the go/no-go decision clearer.

Setting the Broader Business Goals, or Benchmarks, for Accurate Concept Test Scoring:
Once the right set of KPIs have been decided upon, another and bigger challenge awaits. How does a Product Manager decide if the scores coming back from the agency are good, bad, or ugly?

Respondents are often sceptical of new product ideas, especially those that challenge their concept of the status quo or that step outside their personal comfort zone. In addition, they may not be able to visualize the benefits that a new product will provide.

In Jane’s experience, going by absolute scores in such cases is a good way to kill off promising ideas.

Luckily, there is a way out – Historical data or industry knowledge makes a sound starting point. And for new markets without these inputs, the other concepts in the study can be used to judge relative performance and customer acceptance. Accurate benchmarks for the scores help determine if the concept will fare well in market conditions or not.

.Jane, having explained these critical parts of concept testing to Alex, continues:

“You have my full support, Alex – just be sure to choose your KPIs and benchmarks carefully”.
When Alex first considered concept testing his new product, he had no idea it would involve so much thought! Striving not to feel overwhelmed, he reminds himself: first things first. Find a suitable research partner. One that can also give him the most reliable and actionable results.

Authored by Shiva Seth – Manager, Market Research Practice at Absolutdata & Akanksha Mittal – Research Analyst, Market Research Practice at Absolutdata