Have a question about NPD’s research methods? Let us know. Send a question to your NPD account representative, or directly to our Chief Research Officer, Steve Coffey, at steve_coffey@npd.com. We’ll get back to you straight away – and you may see your question in a future Research Edition of NPD Insights. For this edition, we surveyed the research scientists across NPD for the questions they address most frequently.

Question:  Why is NPD so conservative about making changes to the consumer tracking questionnaires?

Answer: The consumer tracking studies conducted by NPD endeavor to size the market and its brand/retailer composition monthly. Market measurement is challenging since we are relying on consumers to recall their purchase behavior. Two factors drive the respondents’ entries.

First is the respondent’s understanding of what is to be recorded. Should, for example, the purchase of crayons be recorded in our toy measurement survey? Second is the respondent’s ability to remember making such a purchase at the time of completing the survey.

The initial screening questionnaires aim to define the product categories in which we are interested. Even small changes to the wording of the screeners have at times resulted in large shifts in the reported volumes. This happens because some proportion of respondents will realize that we mean to include or exclude items they hadn’t previously considered part of the category. When we make a change, then, we are at risk of introducing a disruption in the reported trends as a result of those changing perceptions. By keeping the screening device constant, we are able to more reliably read marketplace trends.

Similarly, the language used, especially the language in the beginning of the survey, may stimulate or fail to stimulate the respondent’s memory about a more casual purchase. There too, the addition of a question, or the addition of an unrelated screening question, can influence the results for other categories.

We utilize a protocol for making changes to the tracking questionnaires that requires testing of wording changes so that we will know in advance whether or not we are at risk of a trend disruption.  With the test results in hand we can take preventative steps to reduce the impact. Trend disruptions still occur creating difficulty for clients. Therefore, we limit changes to only those that are really necessary to improve the data quality, and further accumulate change requests to reduce the number and frequency of such potential disruptions.

Question:  Why do market shares for some brands fluctuate so much from one period to another in the consumer data?

Answer: Assuming that a questionnaire change is not the cause, the most common explanation for fluctuating market shares is smaller sample size. In our ongoing efforts to improve accuracy (reflecting the marketplace) and precision (doing so reliably month after month), we have two very different objectives at play – often working at cross purposes with one another. The number of raw observations is the best predictor of reliability. We recommend exercising caution when the number of raw transactions falls below 200.

For an excellent description of accuracy versus precision, please take a look at Wikipedia’s entry on the topic:  http://en.wikipedia.org/wiki/Accuracy#Accuracy_vs_precision_-_the_target_analogy

Question:  Why is it that trends in the consumer data do not always match the trends reported in the POS data?

Answer: The majority of NPD’s consumer tracking databases are calibrated to known sales levels determined by POS data feeds from retailers. Among the participating retailers, as a group, we conform the consumer data exactly. The POS databases are typically projected to selected retailer channels, groups of like retailers, such as the mass merchant channel. The consumer data, however, represents the complete retail landscape, including non-participating retailers and channels that are not part of the POS offering. Therefore, the two databases, POS and consumer, do not reflect the same defined universe. In most cases the consumer data is the larger of the two universes, including more channels and retailers than the POS offering.