Heavier Responders in Online Survey Research
Professional Panel Management Matters

A Joint Industry Research on Research Study

This article is an extract from a joint industry research paper prepared by NPD, Ipsos Insight, and TNS.

Online research continues to grow, given its benefits of lower cost and faster turnaround than traditional mail-based research methodologies. Many firms, including The NPD Group, have reinvested those savings into larger samples for increased precision in survey results. NPD’s previous paper-and-pencil diary panels, for example, surveyed 13,000 consumers monthly. Today we field more than 150,000 surveys weekly.

To satisfy the growing demand for online research, some other Internet survey providers are inviting their respondents to complete an ever-increasing number of surveys. As a result, concerns have emerged as to whether these practices are having an impact on the business decisions made based on the surveys’ results. 

Our research indicates that more frequent survey takers are not substantially different, and that they represent a very small percentage of a typical study’s sample.

Experienced, professional researchers understand the importance of well-managed samples and panels, and responsibly limit the frequency of survey invitations sent to panelists. But research companies cannot control panelists’ exposure to survey invitations other than the ones they issue themselves. Frequent survey takers make up a portion of any sample. The influence of these “heavy responders” is a function of their importance to the total sample, and the degree to which they provide different results or not. Our research indicates that more frequent survey takers are not substantially different, and that they represent a very small percentage of a typical study’s sample.

Research on Research

Last year, The NPD Group, Ipsos, and TNS, three research industry leaders, formed a cooperative partnership to identify and better understand the behavior of heavy responders and their potential impact on survey results. Senior researchers from each company worked together to design a “research on research” study that would identify and quantify differences in study results related to heavy survey takers. The focus was traditional concept tests, since each of the three firms’ samples are routinely used for such projects. 

A roughly equal number of panelists from each company’s professionally managed sample participated in a number of concept tests across different product categories. Upon completion of the standardized concept surveys, respondents were asked to complete a standardized “Survey Participation” questionnaire covering their personal online survey-taking behaviors. The questionnaire covered surveys delivered by the sponsoring research organizations, as well as those from any other research organization. Questions included survey participation frequency and factors influencing decisions to participate.

How Many Surveys do Respondents Claim to Take?

Approximately 81% of survey responders claimed to have completed ten or fewer surveys from any survey source in the “past 30 days;” the average number was 7.9. The claimed completed average from the sponsoring supplier was 4.0. Each of the participating firms has strict survey frequency controls that keep this figure low.

Heavier responders are somewhat different from lighter responders in terms of their demographics. The very heaviest 3% of responders (those who claimed to have taken 31+ surveys in the past thirty days from any source) were more likely to be men, be part of smaller households, have lower incomes, and/or have lower education levels. This may be initially counter-intuitive -- but since these groups tend to be scarcer in unmanaged online samples, they may be more likely to be invited to participate in projects to achieve demographic quotas. It’s worth noting that this heaviest responder group represents a very small proportion of the participating firms’ returning samples.

Impact of Heavier Responders on Key Measures

Setting aside the demographic differences for the moment, it is important to determine whether heavier responders produce different survey results. Although there are statistically detectable differences between the heaviest and lightest survey participants (heavier responders provided lower purchase intent and value scores on the tested concepts), there are too few in the samples to alter the overall results. Further, there are indeed those demographic differences at play.

The table below provides the details of top two box purchase intent by self-reported survey frequency in the past 30 days. The one group that does stand out as different is that of the very heaviest responders, or those claiming to have taken 31+ surveys in the past 30 days, representing an average of 3% of the responding samples. Some of the variability is explained by the various concepts taken, which may not have been equally distributed across survey-taking classes

Those participating in more surveys are more likely to be doing so for monetary reasons and less likely for non-monetary reasons. "I enjoy doing surveys" was an exception to this rule.

On these professionally managed panels, heavier responders do not meaningfully impact key survey results. The table below summarizes the mean scores (on a five-point scale) for the concepts as assessed by the total sample, those who completed 30 or fewer, 20 or fewer, and 10 or fewer surveys in the past 30 days. Mean scores do not materially change by excluding heavier respondents.

Reasons for Participating

Reasons for participating in survey research also differed by survey participation rate. Those participating in more surveys are more likely to be doing so for monetary reasons and less likely for non-monetary reasons. “I enjoy doing surveys” was an exception to this rule.

Well-Managed Panels

The volume of survey activity on the Internet has risen dramatically due to lower cost per complete, compared to traditional survey methods. Respondents have choices, and some will choose to become more engaged in survey taking than others. That inevitably leads to a class of heavier responders in the national pool of survey takers.

Research professionals understand the importance of sound sample management practices, and the importance of balancing well-crafted incentive systems, survey invitation frequency, and client survey demand. The NPD Group, Ipsos, and TNS exercise considerable restraint on survey frequency, minimizing reliance on the heaviest of survey takers despite higher operational costs. 

A Closing Thank You

Each year NPD conducts several methodology-focused research on research projects to advance our understanding of online research, with the ultimate goal of increasing the quality of the products and services we provide our clients. We were pleased to participate in this joint study and wish to say thanks to our colleagues, Susan Frede of TNS, and Lee Markowitz of Ipsos Insight, who helped make it a success.

Steve Coffey

For more information about how NPD’s research excellence can benefit your company’s decision-making, visit www.npd.com or e-mail contactnpd@npd.com.