US Insights

Sampling best practice déjà vu

Susan Frede

Vice President of Research Methods and Best Practices

Digital 06.14.2016 / 10:45

speech bubble

The same techniques that nonprobability samples use to weight and model data often need to be used on probability samples to account for nonresponse bias

A dozen years ago, a debate raged in the marketing research community over the switch from probability sampling methods such as random digit dialing by telephone (RDD) to nonprobability sampling methods as are typical with online access panels. In the interim years, most marketing research clients moved to online samples, but some marketing researchers—and many public opinion researchers—still cling to probability methods.

However, we now see the quality of probability samples being questioned on a regular, very public basis because of low response rates for RDD. In an interesting twist, the very same techniques that nonprobability samples use to weight and model data now often need to be done on probability samples to account for nonresponse bias.

The Pew Research Center recently published a report entitled, “Evaluating Online Nonprobability Survey," having fielded a survey to nine online nonprobability samples from eight different suppliers. The survey covered 56 measures, including 20 benchmarks, with comparison data obtained from high-quality government research. Results were also compared to Pew’s American Trends Panel (ATP), which was recruited via RDD in early 2014. 

Lightspeed GMI was one of the suppliers in the Pew study, and true to this new world of nonprobability sampling coming full circle, many of Pew’s findings coincide with what we recommend as best-in-class sampling. Our basic premise is to pull an appropriate sample using the right variables for a particular client. Sampling should be customized for each client based on their expressed objectives and needs. In the Pew study, we generally performed in the middle or middle-upper part of the pack. In this case, Pew gave us sampling instructions to control on only age and gender. In hindsight, we should have insisted on additional controls to improve sample quality and performance. Applying all our best practices for nonprobability sampling would have moved us to the top of the pack.

The Lightspeed GMI Sampling Best Practices include:

  1. Understand the objectives and analytic plan for the research and who the end client is so an appropriate sample is built.
  2. Regardless of objectives/analytic plans, at a minimum, balance the outgoing sample on age, gender, region, and income/social class. Additional variables may be needed based on the objectives and analytic plan. 
  3. Be cautious with too much interlocking of quotas. The basic recommendation is to only interlock age and gender. When more is interlocked it creates a level of complexity that is very hard to deliver against. 
  4. Instead of interlocking, if a subgroup is particularly important (e.g., Hispanics, African Americans, young adults, males, etc.) consider splitting the sampling by those subgroups to make sure the subgroup is truly representative. For example, instead of building a single sample two samples might be built—one for males and another for females.
  5. Set survey quotas for key variables. This should generally be age and gender plus one other key variable.
  6. Be cautious with short field periods. At a minimum, we would recommend a two- to three-day field period and for more academic research might go to at least seven days. 
  7. Don’t be afraid to weight data. RIM (random iterative method) weighting is the recommended approach. Before weighting on non-demographic characteristics, think carefully about the implications. 
  8. Remember sample sources aren’t the same, so devise a sampling plan that allows the sample blend to remain fairly consistent for the life of the project. 

Source: Lightspeed

Editor's Notes

A version of this article was originally published on the Lightspeed GMI blog on May 31, 2016. Journalists, to speak with the author, contact us. Follow us on Twitter for the latest news or sign up to receive our email alerts.

Latest Stories

Retailers taking the biggest hit from COVID-19 are largely based in the mall.

Growth benefited from a comparison to very weak growth in the year-ago period when the government shut down.

COVID-19 and its effects on retail dominate discussion of Costco’s future.

Just 24% of US consumers expressed interest in the Apple Credit Card last month.

The Latin American retail sector has outperformed all other categories.

Related Content