Managing Consumer Tracking Studies
By Steve Coffey, Chief Research Officer, The NPD Group, Inc.

Research study design encompasses four very broad content areas:

• The sample
• Data collection – the questionnaire
• Data processing rules/quality control (QC)
• Interpretation

In our last Research Edition, we reviewed NPD’s Data Quality Checking protocol for QC. This time, we’ll cover data collection – the questionnaire.

Experienced researchers know even minor modifications carelessly made to a consumer tracking survey can have significant, destabilizing effects on the trends – even though, with hindsight, this negative impact could have been predicted. Nevertheless, periodic survey changes are necessary in a changing market or to make questionnaire improvements. Such changes can be made safely, but only by learning from the past. NPD’s Questionnaire Change Protocol embodies this learning.

This protocol guarantees when surveys are modified only to achieve clear business objectives, changes are executed with great care and align with our cumulative experience and best practices. The protocol further requires a testing regimen and a comprehensive plan for introducing the new data to our clients. The goals are to make necessary improvements to the surveys while preserving trendability with past data wherever possible, and to minimize the number and frequency of future changes.

The Questionnaire Change Protocol has five key components:

1. Establish business objectives
2. Draft changes – questionnaire review
3. Field test – analysis of test results
4. Implications and Implementation Plan
5. Final approval

Each tracking study is assigned an experienced staff researcher charged with the design and evolution of the underlying methodologies, including the questionnaire instrument. He or she manages the questionnaire change process, taking the business through the protocol.

Establish Business Objectives

Before making any changes, we first list the reasons behind the change, and how the change will benefit our clients. The scope of an industry we track may need to be redefined, for example, to include emerging sub-categories or new retail channels. We may have learned from respondents that an existing question is unclear, causing confusion and misreporting.  Or, perhaps a question is deemed no longer useful or meaningful. Each proposed change is put to the same test: How will our clients benefit from this change to the questionnaire?

Clear business objectives are essential so each prospective change can be evaluated in terms of its ability to bring the work closer to the goal. The first step then is to gather input from Client Development (representing feedback from clients) and others within the organization to articulate and agree upon a comprehensive set of objectives.

Draft Changes – Questionnaire Review

After a first draft of changes has been prepared, fellow researchers provide the essential second (and third and fourth) pair of eyes, to make sure we have not overlooked potential problems prior to testing. Whenever possible, a full review by the Questionnaire Committee takes place before field testing.

Field Test – Analysis of Test Results

Most researchers learn the importance of questionnaire pre-testing early. NPD’s protocol takes that learning a step further, requiring the researchers to “assume nothing.” It demands a full-scale, in field, side-by-side test of the new questionnaire, in direct parallel to the production version of the survey. This provides the researcher with hard data to determine if the changes meet the business objectives, if unanticipated changes are manifested in the collected data, or if what we thought were clear questions turned out to be confusing for panelists. 

Results are analyzed and presented to the Questionnaire Review Committee with a recommendation to approve the new questionnaire or make further changes. 

Implications and Implementation Plan

Once the questionnaire meets the researcher’s approval, the Operations team is notified of the change in preparation for taking the new data through the production process. Subsequent changes to data processing specifications often are required. To the extent further methodology changes are required to accommodate the new data, they are detailed and agreed upon by the Operations team. Expected changes in the data are discussed with the Product Management teams, so Client Development and clients know what to expect, and when.

Final Approval

The final questionnaire, marked with changes, and the analysis of findings are presented to the Questionnaire Review Committee, which formally releases (or doesn’t) the new questionnaire for live production. Controls throughout the production process ensure only approved changes can go live. By production time, only the changes necessary to meet the business objectives will have been made, those changes will have been tested and verified, and the organization – from Operations, to Client Development, to our clients – is aware of and prepared to handle the new information.

Summary

The Questionnaire Change Protocol keeps NPD’s consumer tracking surveys aligned with the changing needs of the many industries we serve. We go to great lengths to ensure only the changes necessary to meet the business objectives are made (thus preserving historical trends wherever possible). We take great care to verify that all changes are made competently and meet the business objectives. Further, all stakeholders, from Operations straight through to our clients, will be prepared for the changes once deployed.