Imagine two groups of people reading about the same candidate for mayor. Half are told nothing unusual about the candidate, while the others learn that an independent organization has accused the candidate of corruption in awarding government contracts. Then everyone gets the same question. How likely are you to vote for that candidate?
Researchers posed that scenario to thousands of people in Brazil, India, Japan, Nigeria, the Philippines and the United States. In every country, the corruption allegation hurt the candidate. The result held across two online survey platforms, even though the people taking those surveys did not always look much like the populations they were meant to represent.
School of Public Policy Dean Gustavo Flores-Macías and his co-authors explored that question in a new study published in Public Opinion Quarterly. They tested two widely used online survey platforms in six countries to see how differences in the people taking the surveys affected the results. “Most of the public opinion research on the strengths and weaknesses of online survey platforms has focused on the U.S. context,” said Flores-Macías. “However, these platforms are increasingly used for cross-national public opinion research, in great part because they allow researchers to collect data quickly and affordably.”
Flores-Macías conducted the research with Dino P. Christenson of Washington University and Sarah E. Kreps and Douglas L. Kriner of Cornell University. They used Morning Consult and Lucid, two platforms that recruit survey respondents differently. Morning Consult uses quotas and post-survey weighting to more closely approximate population benchmarks. Lucid relies on lower-cost opt-in panels and does not use quota-based sampling but can often deliver survey samples at a fraction of the price.
The team surveyed about 1,000 people through each platform in Brazil, India, Japan, Nigeria and the Philippines and roughly 2,000 through each in the United States. They then compared the samples with population benchmarks. Education produced some of the largest gaps. College-educated respondents were significantly overrepresented on both platforms in every country except the United States.
Those disparities matter if researchers want to know precisely what a population thinks. A survey with too many college graduates, for example, may not accurately estimate how common a particular opinion is across an entire country.
The corruption experiment tested something different. Rather than asking researchers to estimate how many people held a particular view, it allowed them to measure whether new information changed people's response.
The samples in different countries sometimes differed significantly from the underlying populations, for example, overrepresenting highly educated respondents. However, the estimated effects of our corruption experiment were very similar across countries and survey platforms.Gustavo A. Flores-Macías Dean, University of Maryland School of Public Policy
“A very interesting finding was the contrast between representativeness and experimental effects,” said Flores-Macías. “The samples in different countries sometimes differed significantly from the underlying populations, for example, overrepresenting highly educated respondents. However, the estimated effects of our corruption experiment were very similar across countries and survey platforms.”
The findings do not mean that an inexpensive online sample can substitute for more rigorous sampling in every study. The researchers found some cross-national differences in the size of the corruption effect, and those differences were sometimes sensitive to the platform being used. They also caution that results from this particular experiment may not carry over to other types of experimental treatments.
For Flores-Macías, choosing a survey sample ultimately comes down to what a researcher is trying to learn. “I hope researchers take away a more nuanced view of convenience samples, especially those working in international or cross-national settings,” he said. “Online convenience samples clearly have limitations, especially if the goal is to estimate precisely the views of an entire population. However, the findings suggest that lower-cost samples can still be very useful for estimating causal effects, including across quite different national contexts.”
