Online versus Telephone Surveys

posted by Bob Mills

Business owners and communication managers are often confronted with the task of conducting research to measure the opinions and attitudes of their customers, members, stakeholders and the public. One of the questions they often ask is, “Can we do an online survey or should we do telephone interviews?”

Among many researchers there is a strongly-held belief that telephone interviews are more reliable than online polls. They cite three strong arguments:

• In a telephone survey, participants are selected in a way that ensures they are a true random sample of the total population;
• The survey size is controllable, allowing for a precise estimate of the reliability of the poll; and,
• Experienced telephone interviewers can clarify questions that respondents struggle to understand, ensuring that everyone’s views are properly heard and understood.

Self-administered online surveys, on the other hand, are deemed to be less reliable because:

• In most cases, participants choose whether or not to participate in the survey – which may skew the results by under-representing those who choose, for whatever reason, not to participate; and,
• Participants who have language barriers or less education may have trouble comprehending written questions.

Proponents of online surveys, however, argue that:

• Online surveys are easier to organize and less expensive to administer;
• With proper methodology, the survey sample, if not random, may be weighted to correct for any bias that shows up in the results;
• Telephone surveys may fail to capture certain demographics like young people and/or cell phone users, who would be more likely to participate in an online survey; and,
• Online surveys are perfect for targeted groups, like employees or association members.

So what’s the truth? Are online surveys more reliable than telephone surveys, or is it the other way around? 

Testing the Truth

In 2014, the Pew Research Centre decided to find out. Using its nationally representative American Trends Panel, Pew conducted a large-scale experiment that tested a telephone survey carried out by an interviewer against a self-administered survey online. A total of 3,003 respondents – randomly assigned to the two groups – were asked the same 60 questions.

The study found some differences in response between these two survey methods, however, they were not typically large. The mean difference was on the order of 5.5 percentage points. Where larger differences did occur, they tended to be focused on questions related to family and social life, with differences of 18 and 14 percentage points respectively.

Online respondents, for example, were more critical of political leaders and less sympathetic to the circumstances of visible minorities than their telephone interview counterparts. In other words, they were more blunt, or perhaps more emotional, in stating their views. However, when the questions focused on simple facts, such as what activities they engaged in yesterday (like reading the newspaper or listening to the TV news), there was no meaningful difference in the results generated by the two survey methods.

Our View

Given the care with which Pew’s panel members are selected, this study may underestimate the differences between the survey methods. Not all online polls will be equally successful in gathering respondents who reflect the makeup of the total population. At the very least, the results of any online survey should be carefully examined for representational bias and the results should be viewed as not provably reliable for decision-making purposes.

However, online surveys may well be a perfect tool for gathering input from highly-targeted groups, like company employees (particularly white collar workers and professionals) and from members of professional associations and clubs. By their nature, these groups share many common experiences and interests and there is a stronger likelihood that an online survey, with appropriate participation rates, will adequately capture the sentiment of the total population. Plus, surveys of this type are easy to organize and the data is automatically captured and processed by online survey software.

In the case where public opinions are solicited on social issues, we expect that both telephone and online methods will generate their own bias. With a live interviewer, many people will mute their more controversial opinions, whereas online they may feel freer to express their frustrations and vent their feelings. In a situation like that, one could easily make an argument for using telephone surveys AND online surveys to get a more balanced result.