Advancing Accuracy and Precision with Calibration

By Ash Dhupar,
Vice President, Research Sciences
The NPD Group, Inc.

NPD offers both point-of-sale (POS) and consumer-reported information for many of the industries we serve. We combine them in a process called calibration, which compensates for biases in consumer reporting, resulting in much more precise and accurate data than is otherwise possible.    

Why calibrate?
Errors are inherent in all consumer research. NPD’s Research Sciences team is devoted to minimizing such errors through methodological design. Here we will examine some common types of consumer research error and outline how our most powerful tool, calibration, reduces these biases.

The first type of error we encounter is called random sampling error (RSE), which surfaces whenever a sample is used rather than measuring an entire population. It can result in under- or over-estimating a measurement, and varies randomly from one time period to the next, by brand, by category, or other similar view. RSE cannot be eliminated entirely, but can be significantly reduced by using very large samples or by fixing estimates to the true population mean. In the online world, we are able to use extremely large samples that substantially reduce RSE.

Non-random sampling error (NRSE) can be sample-related, resulting from deviations (intentional or otherwise) in the reporting from a random probability sample. For example, contemporary sampling favors respondents who are more likely to respond to a survey. In our January Research Edition of the NPD Insights newsletter, NPD’s Dr. Mark Kinnucan discussed the potential impact of sampling only from an online population. Unlike random sampling error, sampling design issues have a consistent impact on the estimates over time.

Non-random sampling errors also include biases not related to the sample itself, such as:

• The influence of the questionnaire on the respondent
• Panelists’ ability to recall some purchases over others
• The effects of advertising and awareness resulting in fewer “don’t know” responses 

Most non-sampling error is systematic and consistent over time. NPD’s calibration process significantly reduces this measurement error, giving us higher accuracy and reliability in our reporting.

Identifying biases
Coverage is defined as the degree to which consumer data accurately estimates POS actual sales (the true population mean). Sales estimates within a defined category of retailers are compared to actual POS data. Coverage of 100% means the consumer data estimate is exactly on target; above 100% indicates consumer overstatement, and below 100%, understatement. Understanding what drives coverage deviations helps us determine which variables should be used during the calibration process.

The Research Sciences teams at NPD have identified several key drivers of coverage across the many categories we’ve studied:

Price: Our observed coverage of low-priced items is low, and coverage of high-priced items is high. We can deduce then that price is related to a panelist’s ability to recall a purchase. People more adequately remember purchasing expensive items, while inexpensive items are less memorable. We therefore make different adjustments by price band in our calibration system.

Seasonal influence: During high-volume purchasing seasons, such as the October through December holiday season, consumers have better recall because of their high involvement with gift purchasing.  However, we also see coverage declines toward the close of holiday shopping, as the number of items purchased rises, making it easier to forget an item here or there. Calibration systems, therefore, must be sensitive to seasonal and month-to-month fluctuations in coverage.

Retailer type: Coverage varies by retailer and shopping occasion. Consumers readily recall purchasing a toy in a toy specialty retailer (coverage is a remarkable 100% in such circumstances), but may not be as likely to recall purchasing a small toy as part of a larger shopping occasion in, say, a drug store. Retail channel, therefore, is an important dimension in the calibration process.

Social Desirability:  Respondents may be more inclined to remember or to report, for example, educational toys or very trendy items.

Compensating for biases
The good news is these kinds of biases can be reliably measured. NPD’s unique calibration methodology does so in two phases: First, long-term, consistent bias is identified across product, brand, and channels, and then corrected. The second pass shores up residual month-by-month deviations.

Analysis over long time periods reveals consistent patterns of bias. Historical estimates of coverage are calculated and adjusted. This approach allows measurement and adjustment for biases at a very granular level of detail.

In the chart above, for example, Brand X is over-reported on average by about 50%, evidenced by the higher Raw Consumer line compared to the POS line. A consistent adjustment is applied to reduce the over-reporting, resulting in a much closer fit between Adjusted Consumer and POS. In a later step, month-by-month adjustments are applied, but importantly, we’ve learned that this brand has a measurable bias, and we can use that measurement and the corresponding correction on the target brand, and elsewhere as well.

Adjustments for Non-Participating Retailers – Borrowing
Direct adjustments, such as the above, are only possible when aligned POS data is available. In the previous example, the consumer estimates were reduced to more closely align with POS. Since POS data is not available from all retailers, we make the reasonable assumption that if we must reduce (or increase) the consumer estimate among those retailers that do provide POS data, we must reduce (or increase) the consumer estimate by a like amount for those retailers that do not provide POS data. In this way, the data essentially tells us how it needs to be adjusted.

Sample size and calibration
Sample size plays an important role in determining the level to which we are able to perform calibration.  We need to strike a balance between capturing bias nuances, yet have reliable measurements of that bias. Reliable measurement requires sufficient observations in the consumer data to read actual bias over random sampling error.

We studied a broad number of brands and categories, assessing the correlation between consumer and POS trends. We found, as expected, a direct link between the number of consumer transactions used in the estimate, and the alignment with POS.  

Further analysis conducted via bootstrapping measured the random error associated with number of transactions. This analysis, combined with the above, helped us establish minimum sample thresholds for calibration.

Pulling it together
Having identified the adjustments necessary at a brand, category, channel, price-point, or other level, the adjustments are combined in an iterative process to bring all adjustments into alignment. It is this multivariate adjustment that is applied to the data, prior to the final monthly “truing-up” of the final estimates. In practice, the final monthly adjustments are quite small, typically ranging from 0% to 5%.

Again, we use long time periods to develop adjustments at granular levels, and then apply those adjustments to all the consumer data. Then, the final “current month” is tightened up should any further adjustments reveal themselves as necessary.

In closing
We employ a great deal of rigor in the methodology designs of both our consumer and POS databases. In those cases where both systems are measuring the same industry, we have the special opportunity to calibrate the consumer data with POS, resulting in more accurate and reliable estimates from the consumer database than is otherwise possible from consumer-based research alone.