Using Manufacturing Quality Control Checks in Marketing Research
By Steve Coffey, Chief Research Officer, The NPD Group, Inc.

As part of NPD’s Quality First initiative, our Research Sciences team has developed and is expanding formal protocols for each of the steps in the research process: sampling, data collection, data cleaning, tabulation and analysis. The protocols – with quality control checks at each step – articulate our accumulated knowledge of best practices based on experiences among the researchers, external research, and original research and development we have conducted. They establish the standard operating procedure that must be followed. As new NPD protocols are developed, they are reviewed by the NPD Research Council, a decision-making body composed of the senior research leadership within NPD that meets monthly to make policy, review projects and approve major research project designs. Once these protocols are reviewed, they become policy. The quality checks already in place at each step in our production processes are also reviewed, and, where necessary, improved.

One of these new quality control (QC) protocols addresses the final QC checks of finished databases to reduce errors. After all of the effort involved in the release of a single NPD database, it is heartbreaking to us when a client calls to report an error that we missed. To reduce that possibility, we developed a Final QC Protocol. At the core of the protocol is Edwards Deming’s work on statistical quality control. This methodology is especially well suited to NPD’s tracking businesses, both POS and consumer, since we have consistent market measurements over time. Here’s how we have applied it to our work:

Let’s assume we are evaluating the January 2006 unit sales estimate for Brand X digital still cameras in Retailer Y. We assemble the prior 24 months of unit sales estimates for Brand X/Retailer Y, and then compute the monthly change versus the prior year for each of the preceding twelve months. The goal of this step is to understand the typical year-over-year variation by month for this product. To get that, we compute the mean change versus year ago, and the standard deviation of those changes. The standard deviation tells us what kind of month-to-month variation has been observed, or what is normal for this brand at this retailer. By definition, we would expect 95% of the cases would fall within roughly two standard deviations. Therefore, if the month being evaluated (the current month) falls more than two standard deviations away from the historic mean, we have a suspicious estimate. Said another way, we use these statistics to set up a range of reasonableness for the current month’s estimate. When we identify a suspicious estimate, there are two possible explanations: 1) The marketplace has taken a sudden turn, perhaps due to a special promotion or in-market situation, or 2) there is an error somewhere in our process.

The table below shows the January 2006 sales estimate for this technology item is 250 units. Is that reasonable? Probably not, it turns out. The average percent change over the prior 12 months was +29%, with a standard deviation of 9%. The increase of 67%, then, is 4.2 standard deviations outside the mean – highly unlikely, given historical patterns. It suggests that either the item was unusually promoted, or there may be an error in the production of the estimate.

To examine the formulae used in an Excel spreadsheet, please click here to download a copy.