As one of the leading polls in North Carolina politics and public policy, Civitas Poll releases around 10 statewide polls per year along with special topic polls (such as our annual school choice poll or polling of specific North Carolina legislative or congressional districts).
The 2016 presidential election left many people skeptical about the accuracy of public opinion polling – and understandably so. Most polls predicted Democratic presidential nominee Hillary Clinton as the winner, a fact that was widely publicized by the mainstream media leading up to election day.
Since President Donald Trump’s poll-defying victory, experts in the polling industry have put forth several theories about why the polls got it so wrong. Regardless of the reason, public confidence in polling was undoubtedly shaken.
No poll is perfect. By their very nature, polls are estimates. They use a sample of the population to make a “best guess” about the opinion of the larger target population. That being said, polls can be very useful tools in understanding public perception. Civitas describes its polling as a “snapshot of public opinion” at a given time.
Before things ramp up for the 2020 election, we’d like to take the opportunity to explain how our poll works – and thus enhance your understanding of poll results and how to interpret them.
Poll structure
The monthly and special Civitas Polls are conducted by professional polling organizations, including Harper Polling and SurveyUSA. Civitas Polls are typically live-call polling, meaning they are conducted via telephone with a poll worker administering the survey. Around 65 percent of respondents come from landlines while 35 percent are from cell phones.
Civitas Polls usually have a sample size of around 500 self-identified likely voters. This is a distinction of the Civitas Poll, since several other North Carolina polls use registered voters or some other population criteria. The poll sample consists only of individuals who complete the entire survey. Per industry standards, Civitas Polls have a confidence level of 95 percent and a margin of error is typically around 4 percent – two important poll elements discussed in detail below.
Statistical considerations
At face value, polls only convey with certainty the sentiment of the specific individuals who took the survey. But the larger value of polling is the insight it provides into the feelings of the larger population. However, we can’t assume that the population surveyed will exactly match the larger population every time. For that reason, polls rely on statistical tools to communicate what the sample results can tell us about the larger population’s views.
Margin of error (MOE)
First is the poll’s margin of error. The MOE provides the range estimate for where the true population’s results will most likely fall. The concept is best explained through an example:
If a fictional poll reports that 55 percent of North Carolinians prefer Western barbecue over Eastern barbecue, and the poll has a margin of error of 4 percent, the more accurate interpretation is that 55 percent is the best guess for the true preference of the state, but the actual preference is probably between 51 percent and 59 percent (Still a majority, in this fictional poll, since Western is clearly the superior choice).
As mentioned, the margin of error for Civitas Polls is usually around 4 percent, although it varies by poll. The MOE is always stated in Civitas Poll press releases and documents.
Confidence level
The margin of error uses the sample results to provide a likely range for the true population result. The key word in that sentence, however, is “likely.” The confidence level communicates the likelihood that the population result falls within the bounds of the margin of error range (known as a confidence interval). Let’s go back to the barbecue example:
Twenty identical polls are conducted at the same time, each with a 95 percent confidence level and a 4 percent margin of error. Nineteen of them demonstrate that Western barbecue obtains between 51 and 59 percent of respondent’s preference, but one poll shows Western barbecue with only 48 percent of the votes. The results of the one poll don’t undermine the validity of the other polls, nor is that an indication that the outlying poll was poorly conducted. It is simply the result of random chance; the one sample of survey respondents happened to skew more towards an Eastern preference than the true state population would, due to nothing more than random chance.
Per industry standard, Civitas Polls have a 95 percent confidence level, meaning that if we conducted the same poll 100 times, 95 of those polls would produce a result reflecting the “true” opinion of North Carolina’s population, within the margin of error.
Interpretation implications
What do these technical elements of polling mean for the interpretation of poll results?
First, it is important to keep the polled population in mind when interpreting Civitas Polls. As mentioned, most Civitas Polls use a sample of “likely voters.” This methodological choice has advantages and disadvantages. By focusing on likely voters, Civitas Polls are more likely to reflect election outcomes than other polling methodology. It also best helps elected officials ascertain the views of the civically-engaged segment of the state population. One could make a case, however, that likely voters may feel differently about certain public policy issues than the general population of the state. Occasionally, some special topic Civitas Polls use other criteria for sampling, such as “registered voters,” to define the polled population. Whether using polls by Civitas or other organizations, responsible poll users should identify the population which the poll represents.
The margin of error means that neck-and-neck poll results should be carefully approached. This is especially common for election polls, which often result with candidates within a point or two of one another. Such scenarios are virtually a statistical tie. Conversely, if the gap between candidates or poll options is outside of the margin of error, it provides more compelling evidence for the population’s true feelings.
The confidence level can help explain poll results that seem to be an anomaly, as well. Occasionally, a poll will – by random chance – produce a non-accurate result. Repetition of questions can help determine the population’s true feelings; producing similar results multiple times help support the validity of the results. Public opinion can change, however, both gradually or suddenly, so the confidence level is only one piece of the puzzle.
If all this sounds confusing, that’s because polls are tools that can be very useful but must also be utilized carefully. The full Civitas Poll results are always available, in both the toplines and the crosstabs. But, for the latest poll analysis, be sure to subscribe to our poll email list and view our content at www.nccivitas.org/polling.
In the next installment of this series, we’ll look at non-statistical factors that can influence a poll’s accuracy and how to account for them in your interpretation of poll results.
