Good Research: How to Know It When You See It

By J.D. Deitch, Ph.D.
Analytic Director, Games and Software
The NPD Group, Inc.

The need to understand what consumers do and what they want drives the $22 billion market research industry – and the stakes are high. Simply put, good research can make your new product launch a huge success. Bad research can mean your multi-million dollar advertising campaign is a disaster.

Whether you are a manufacturer, marketer, retailer, distributor, ad agency, or another link in the chain of service to consumers, you most likely rely heavily on market research results to make better business decisions.

So how can you tell the difference? How can you be confident that your consumer data reflects the marketplace? How can you make sure you are getting good research?

The Four Pillars of Good Research
While designing high quality research requires a combination of specialized training, skills, and experience, judging research quality is more accessible and straightforward. We strongly believe the following four elements are essential to insightful analysis and good consumer-based research.

Research can’t help you if you don’t know what the numbers “mean.” So measure exactly what it is you need to succeed. Take sales and market share data for instance. Are you working with a precise definition for the category you’re tracking – what is included, and what is not? What channels of distribution, or groups of retailers, do you need to know about? Are sales to consumers sufficient, or do you need small business and institutional sales as well? What geography are you interested in – total United States, the lower 48 states, all of North America, the international market? Do you want total retail sales, including price discounts, promotions, mail-in rebates, tax, shipping – or is list price sufficient? Knowing exactly what is being measured – what’s in and what’s out – is critical to solid data and market research results.   

The instances where it’s both possible and practical to collect data on an entire population are few and far between. That’s why we almost always rely on samples as the basis of our research. But all samples are not created equal, and it’s important to know the size, composition, and handling of that sample to understand potential pitfalls. The size of the sample gives you a measure of the precision of your estimates: bigger samples have lower margins of error than smaller ones. Proper sample composition ensures that the information collected accurately represents the larger population. Finally, knowing how the data have been handled and collected is also important for understanding bias. Online surveys, for example, are very cost-effective, but the demographics may be skewed in a way that has a bearing on your numbers. Addressing these criteria can result in tradeoffs – and often they’re tradeoffs related to cost. For example, all things being equal, larger and more representative samples yield more accurate and reliable results, but they cost more to collect. Your market research partner should be able to help you understand these tradeoffs and recommend a course of action that meets your needs.

So let’s say you’ve got your results. You’ve used precise, well-defined measures and you were fortunate to have a large and representative sample. How are those raw inputs transformed into final estimates? Are sample projection methods reliable, or are they prone to bias? Are the strengths of multiple sources being used fully to improve the reliability of the final estimate? The statistical procedures used to project a sample to a known population (e.g., the total number of individuals in the U.S. aged 18 and over) should be robust enough to yield a representative view of the population, and to control demographic biases that impact the behavior you’re observing. For example, if you study the toy purchasing behavior only of single adults with no kids, you’re going to get a very lopsided view of the toys industry!

The real value of consumer research is unlocked when it is carefully designed by people with proven expertise and deep proficiency. While new technologies may seem to make it look easy, the proof is in the pudding. Is the firm known for its research experience and heritage? Is the market research partner well-qualified and up to the task? Are experienced, trained researchers making the methodology design decisions? Are they available to answer your methodology questions completely and transparently? Are they available to defend their work? If the answer to any of these questions is no, think twice before engaging it. Literally thousands of methodology decisions are made to get a tracking system up and running – and it just isn’t practical for you, as a customer, to review every single one. It is therefore essential that you have confidence in the teams making those decisions on your behalf. Just as consumers favor brands they trust, it is just good business sense to incorporate the reputation, transparency and dependability of the provider as an additional input.

How does NPD stack up?

We strive to provide our clients with the critical facts behind the measures, sources, and methods we employ, as well as a trusted perspective on what is selling, where it is selling, who is buying, and why. When we start tracking an industry, we start with a standardized, well-researched data model honed by years of experience and best practices. We then optimize our methodology to suit the differences of category. And don’t let our “age” fool you – we are constantly looking for new ideas and ways to improve our methods for accurate estimates in all of your business climates.