A dozen years ago, a debate raged in the marketing research community over the switch from probability sampling methods such as random digit dialing by telephone (RDD) to nonprobability sampling methods as are typical with online access panels. In the interim years, most marketing research clients moved to online samples, but some marketing researchers—and many public opinion researchers—still cling to probability methods.
However, we now see the quality of probability samples being questioned on a regular, very public basis because of low response rates for RDD. In an interesting twist, the very same techniques that nonprobability samples use to weight and model data now often need to be done on probability samples to account for nonresponse bias.
The Pew Research Center recently published a report entitled, “Evaluating Online Nonprobability Survey," having fielded a survey to nine online nonprobability samples from eight different suppliers. The survey covered 56 measures, including 20 benchmarks, with comparison data obtained from high-quality government research. Results were also compared to Pew’s American Trends Panel (ATP), which was recruited via RDD in early 2014.
Lightspeed GMI was one of the suppliers in the Pew study, and true to this new world of nonprobability sampling coming full circle, many of Pew’s findings coincide with what we recommend as best-in-class sampling. Our basic premise is to pull an appropriate sample using the right variables for a particular client. Sampling should be customized for each client based on their expressed objectives and needs. In the Pew study, we generally performed in the middle or middle-upper part of the pack. In this case, Pew gave us sampling instructions to control on only age and gender. In hindsight, we should have insisted on additional controls to improve sample quality and performance. Applying all our best practices for nonprobability sampling would have moved us to the top of the pack.
The Lightspeed GMI Sampling Best Practices include:
- Understand the objectives and analytic plan for the research and who the end client is so an appropriate sample is built.
- Regardless of objectives/analytic plans, at a minimum, balance the outgoing sample on age, gender, region, and income/social class. Additional variables may be needed based on the objectives and analytic plan.
- Be cautious with too much interlocking of quotas. The basic recommendation is to only interlock age and gender. When more is interlocked it creates a level of complexity that is very hard to deliver against.
- Instead of interlocking, if a subgroup is particularly important (e.g., Hispanics, African Americans, young adults, males, etc.) consider splitting the sampling by those subgroups to make sure the subgroup is truly representative. For example, instead of building a single sample two samples might be built—one for males and another for females.
- Set survey quotas for key variables. This should generally be age and gender plus one other key variable.
- Be cautious with short field periods. At a minimum, we would recommend a two- to three-day field period and for more academic research might go to at least seven days.
- Don’t be afraid to weight data. RIM (random iterative method) weighting is the recommended approach. Before weighting on non-demographic characteristics, think carefully about the implications.
- Remember sample sources aren’t the same, so devise a sampling plan that allows the sample blend to remain fairly consistent for the life of the project.